Sep 09 02:30:11.746107 ip-10-0-137-37 systemd[1]: Starting Kubernetes Kubelet... Sep 09 02:30:12.136802 ip-10-0-137-37 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 02:30:12.136802 ip-10-0-137-37 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 09 02:30:12.136802 ip-10-0-137-37 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 02:30:12.136802 ip-10-0-137-37 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 09 02:30:12.136802 ip-10-0-137-37 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 02:30:12.137468 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.136828 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 09 02:30:12.145183 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145157 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 02:30:12.145183 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145180 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 02:30:12.145183 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145184 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 02:30:12.145183 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145187 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 02:30:12.145183 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145190 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 02:30:12.145183 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145193 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 02:30:12.145407 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145196 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 02:30:12.145407 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145200 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 02:30:12.145407 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145203 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 02:30:12.145407 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145205 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 02:30:12.145407 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145208 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 02:30:12.145407 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145211 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 02:30:12.145407 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145214 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 02:30:12.145407 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145217 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 02:30:12.145407 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145222 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 02:30:12.145407 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145225 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 02:30:12.145407 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145228 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 02:30:12.145407 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145234 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 02:30:12.145407 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145238 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 02:30:12.145407 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145241 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 02:30:12.145407 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145245 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 02:30:12.145407 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145248 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 02:30:12.145407 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145251 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 02:30:12.145407 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145253 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 02:30:12.145407 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145257 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 02:30:12.145877 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145260 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 02:30:12.145877 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145266 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 02:30:12.145877 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145270 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 02:30:12.145877 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145274 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 02:30:12.145877 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145277 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 02:30:12.145877 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145280 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 02:30:12.145877 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145283 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 02:30:12.145877 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145285 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 02:30:12.145877 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145288 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 02:30:12.145877 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145291 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 02:30:12.145877 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145294 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 02:30:12.145877 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145297 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 02:30:12.145877 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145299 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 02:30:12.145877 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145302 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 02:30:12.145877 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145309 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 02:30:12.145877 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145313 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 02:30:12.145877 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145316 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 02:30:12.145877 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145319 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 02:30:12.145877 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145322 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 02:30:12.145877 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145324 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 02:30:12.146371 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145327 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 02:30:12.146371 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145329 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 02:30:12.146371 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145332 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 02:30:12.146371 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145334 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 02:30:12.146371 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145337 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 02:30:12.146371 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145340 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 02:30:12.146371 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145345 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 09 02:30:12.146371 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145347 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 02:30:12.146371 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145350 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 02:30:12.146371 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145352 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 02:30:12.146371 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145355 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 02:30:12.146371 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145358 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 02:30:12.146371 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145361 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 02:30:12.146371 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145363 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 02:30:12.146371 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145366 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 02:30:12.146371 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145368 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 02:30:12.146371 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145371 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 02:30:12.146371 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145374 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 02:30:12.146371 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145376 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 02:30:12.146847 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145381 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 02:30:12.146847 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145384 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 02:30:12.146847 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145387 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 02:30:12.146847 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145390 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 02:30:12.146847 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145393 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 02:30:12.146847 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145396 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 02:30:12.146847 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145398 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 02:30:12.146847 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145403 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 02:30:12.146847 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145406 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 02:30:12.146847 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145409 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 02:30:12.146847 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145413 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 02:30:12.146847 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145418 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 02:30:12.146847 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145421 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 02:30:12.146847 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145424 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 02:30:12.146847 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145427 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 02:30:12.146847 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145430 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 02:30:12.146847 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145436 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 02:30:12.146847 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145439 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 02:30:12.146847 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145443 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 02:30:12.146847 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145446 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 02:30:12.147376 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145450 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 02:30:12.147376 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145452 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 02:30:12.147376 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145458 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 02:30:12.147376 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145461 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 02:30:12.147376 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145465 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 02:30:12.147376 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145468 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 02:30:12.147376 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145471 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 02:30:12.147376 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145474 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 02:30:12.147376 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145477 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 02:30:12.147376 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145480 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 02:30:12.147376 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145483 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 02:30:12.147376 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145486 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 02:30:12.147376 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145489 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 02:30:12.147376 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145492 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 02:30:12.147376 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.145495 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 02:30:12.147376 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146628 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 02:30:12.147376 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146635 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 02:30:12.147376 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146639 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 02:30:12.147376 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146642 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 02:30:12.147841 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146646 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 02:30:12.147841 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146649 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 02:30:12.147841 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146652 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 02:30:12.147841 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146655 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 02:30:12.147841 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146658 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 02:30:12.147841 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146661 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 02:30:12.147841 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146667 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 02:30:12.147841 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146670 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 02:30:12.147841 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146672 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 02:30:12.147841 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146675 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 02:30:12.147841 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146678 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 02:30:12.147841 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146681 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 02:30:12.147841 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146683 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 02:30:12.147841 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146686 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 02:30:12.147841 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146689 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 02:30:12.147841 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146691 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 02:30:12.147841 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146694 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 02:30:12.147841 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146697 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 02:30:12.147841 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146700 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 02:30:12.147841 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146705 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 02:30:12.148339 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146708 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 02:30:12.148339 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146710 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 02:30:12.148339 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146713 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 02:30:12.148339 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146716 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 02:30:12.148339 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146719 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 02:30:12.148339 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146723 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 02:30:12.148339 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146725 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 02:30:12.148339 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146728 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 02:30:12.148339 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146730 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 02:30:12.148339 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146733 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 02:30:12.148339 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146736 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 02:30:12.148339 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146741 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 02:30:12.148339 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146743 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 02:30:12.148339 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146746 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 02:30:12.148339 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146749 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 02:30:12.148339 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146752 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 02:30:12.148339 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146757 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 02:30:12.148339 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146761 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 02:30:12.148339 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146764 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 02:30:12.148339 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146767 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 02:30:12.148850 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146769 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 02:30:12.148850 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146773 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 02:30:12.148850 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146776 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 02:30:12.148850 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146782 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 02:30:12.148850 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146784 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 02:30:12.148850 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146787 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 02:30:12.148850 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146790 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 02:30:12.148850 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146793 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 02:30:12.148850 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146796 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 02:30:12.148850 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146800 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 02:30:12.148850 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146804 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 02:30:12.148850 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146807 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 02:30:12.148850 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146810 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 02:30:12.148850 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146826 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 02:30:12.148850 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146829 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 02:30:12.148850 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146835 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 02:30:12.148850 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146838 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 02:30:12.148850 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146840 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 02:30:12.148850 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146843 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 02:30:12.149318 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146846 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 02:30:12.149318 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146849 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 02:30:12.149318 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146852 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 02:30:12.149318 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146854 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 02:30:12.149318 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146857 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 02:30:12.149318 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146860 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 02:30:12.149318 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146863 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 02:30:12.149318 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146865 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 02:30:12.149318 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146868 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 02:30:12.149318 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146873 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 02:30:12.149318 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146876 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 02:30:12.149318 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146878 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 02:30:12.149318 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146881 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 02:30:12.149318 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146884 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 02:30:12.149318 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146887 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 02:30:12.149318 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146889 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 02:30:12.149318 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146893 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 02:30:12.149318 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146898 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 02:30:12.149318 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146902 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 02:30:12.149780 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146905 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 02:30:12.149780 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146910 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 02:30:12.149780 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146913 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 02:30:12.149780 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146915 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 02:30:12.149780 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146918 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 02:30:12.149780 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146920 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 02:30:12.149780 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146923 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 02:30:12.149780 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146925 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 02:30:12.149780 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146928 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 02:30:12.149780 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146931 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 02:30:12.149780 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146934 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 02:30:12.149780 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146937 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 02:30:12.149780 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146939 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 02:30:12.149780 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146942 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 09 02:30:12.149780 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146947 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 02:30:12.149780 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146950 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 02:30:12.149780 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.146953 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 02:30:12.149780 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147561 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 02:30:12.149780 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147566 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 02:30:12.149780 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147572 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 02:30:12.150284 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147575 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 02:30:12.150284 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147577 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 02:30:12.150284 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147580 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 02:30:12.150284 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147583 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 02:30:12.150284 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147586 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 02:30:12.150284 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147591 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 02:30:12.150284 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147594 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 02:30:12.150284 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147598 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 02:30:12.150284 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147601 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 02:30:12.150284 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147606 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 02:30:12.150284 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147610 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 02:30:12.150284 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147616 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 02:30:12.150284 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147619 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 02:30:12.150284 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147622 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 02:30:12.150284 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147626 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 02:30:12.150284 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147628 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 02:30:12.150284 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147631 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 02:30:12.150284 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147634 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 02:30:12.150284 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147637 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 02:30:12.150750 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147640 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 02:30:12.150750 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147643 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 02:30:12.150750 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147645 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 02:30:12.150750 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147648 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 02:30:12.150750 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147654 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 02:30:12.150750 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147657 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 02:30:12.150750 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147660 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 02:30:12.150750 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147664 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 02:30:12.150750 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147666 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 09 02:30:12.150750 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147669 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 02:30:12.150750 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147672 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 02:30:12.150750 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147675 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 02:30:12.150750 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147677 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 02:30:12.150750 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147680 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 02:30:12.150750 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147683 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 02:30:12.150750 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147686 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 02:30:12.150750 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147691 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 02:30:12.150750 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147694 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 02:30:12.150750 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147697 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 02:30:12.150750 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147700 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 02:30:12.151272 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147702 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 02:30:12.151272 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147705 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 02:30:12.151272 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147709 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 02:30:12.151272 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147711 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 02:30:12.151272 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147714 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 02:30:12.151272 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147717 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 02:30:12.151272 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147719 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 02:30:12.151272 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147725 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 02:30:12.151272 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147727 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 02:30:12.151272 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147730 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 02:30:12.151272 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147732 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 02:30:12.151272 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147735 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 02:30:12.151272 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147738 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 02:30:12.151272 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147741 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 02:30:12.151272 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147743 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 02:30:12.151272 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147747 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 02:30:12.151272 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147749 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 02:30:12.151272 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147752 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 02:30:12.151272 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147755 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 02:30:12.151272 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147757 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 02:30:12.151838 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147762 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 02:30:12.151838 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147765 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 02:30:12.151838 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147768 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 02:30:12.151838 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147770 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 02:30:12.151838 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147773 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 02:30:12.151838 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147776 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 02:30:12.151838 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147778 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 02:30:12.151838 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147781 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 02:30:12.151838 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147784 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 02:30:12.151838 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147786 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 02:30:12.151838 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147789 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 02:30:12.151838 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147792 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 02:30:12.151838 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147795 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 02:30:12.151838 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147800 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 02:30:12.151838 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147803 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 02:30:12.151838 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147806 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 02:30:12.151838 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147808 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 02:30:12.151838 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147811 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 02:30:12.151838 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147826 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 02:30:12.151838 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147828 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 02:30:12.152335 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147833 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 02:30:12.152335 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147836 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 02:30:12.152335 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147839 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 02:30:12.152335 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147842 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 02:30:12.152335 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147845 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 02:30:12.152335 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147854 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 02:30:12.152335 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147857 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 02:30:12.152335 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147860 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 02:30:12.152335 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147863 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 02:30:12.152335 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147866 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 02:30:12.152335 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147869 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 02:30:12.152335 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147872 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 02:30:12.152335 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147875 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 02:30:12.152335 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147879 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 02:30:12.152335 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147882 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 02:30:12.152335 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147885 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 02:30:12.152335 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.147887 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 02:30:12.152335 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148119 2577 flags.go:64] FLAG: --address="0.0.0.0" Sep 09 02:30:12.152335 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148272 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 09 02:30:12.152335 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148283 2577 flags.go:64] FLAG: --anonymous-auth="true" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148288 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148293 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148296 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148301 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148308 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148311 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148315 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148318 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148322 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148325 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148328 2577 flags.go:64] FLAG: --cgroup-root="" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148331 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148334 2577 flags.go:64] FLAG: --client-ca-file="" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148337 2577 flags.go:64] FLAG: --cloud-provider="external" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148341 2577 flags.go:64] FLAG: --cluster-dns="[]" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148347 2577 flags.go:64] FLAG: --cluster-domain="" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148350 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148353 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148357 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148360 2577 flags.go:64] FLAG: --container-log-max-files="5" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148364 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148367 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 09 02:30:12.152902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148371 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148374 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148377 2577 flags.go:64] FLAG: --contention-profiling="false" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148381 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148384 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148387 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148392 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148399 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148402 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148405 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148410 2577 flags.go:64] FLAG: --enable-load-reader="false" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148415 2577 flags.go:64] FLAG: --enable-server="true" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148419 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148426 2577 flags.go:64] FLAG: --event-burst="100" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148429 2577 flags.go:64] FLAG: --event-qps="50" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148434 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148439 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148442 2577 flags.go:64] FLAG: --eviction-hard="" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148453 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148456 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148459 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148464 2577 flags.go:64] FLAG: --eviction-soft="" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148467 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148471 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148476 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 09 02:30:12.153466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148481 2577 flags.go:64] FLAG: --experimental-mounter-path="" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148486 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148490 2577 flags.go:64] FLAG: --fail-swap-on="true" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148495 2577 flags.go:64] FLAG: --feature-gates="" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148499 2577 flags.go:64] FLAG: --file-check-frequency="20s" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148502 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148506 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148509 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148512 2577 flags.go:64] FLAG: --healthz-port="10248" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148515 2577 flags.go:64] FLAG: --help="false" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148519 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-137-37.ec2.internal" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148522 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148525 2577 flags.go:64] FLAG: --http-check-frequency="20s" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148528 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148532 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148536 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148539 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148543 2577 flags.go:64] FLAG: --image-service-endpoint="" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148546 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148548 2577 flags.go:64] FLAG: --kube-api-burst="100" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148551 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148554 2577 flags.go:64] FLAG: --kube-api-qps="50" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148557 2577 flags.go:64] FLAG: --kube-reserved="" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148560 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 09 02:30:12.154122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148563 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148566 2577 flags.go:64] FLAG: --kubelet-cgroups="" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148569 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148572 2577 flags.go:64] FLAG: --lock-file="" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148575 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148578 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148581 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148587 2577 flags.go:64] FLAG: --log-json-split-stream="false" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148591 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148594 2577 flags.go:64] FLAG: --log-text-split-stream="false" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148597 2577 flags.go:64] FLAG: --logging-format="text" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148600 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148604 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148607 2577 flags.go:64] FLAG: --manifest-url="" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148610 2577 flags.go:64] FLAG: --manifest-url-header="" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148615 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148618 2577 flags.go:64] FLAG: --max-open-files="1000000" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148626 2577 flags.go:64] FLAG: --max-pods="110" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148629 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148632 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148636 2577 flags.go:64] FLAG: --memory-manager-policy="None" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148639 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148643 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148646 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148649 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 09 02:30:12.154703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148658 2577 flags.go:64] FLAG: --node-status-max-images="50" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148661 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148664 2577 flags.go:64] FLAG: --oom-score-adj="-999" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148668 2577 flags.go:64] FLAG: --pod-cidr="" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148671 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148677 2577 flags.go:64] FLAG: --pod-manifest-path="" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148680 2577 flags.go:64] FLAG: --pod-max-pids="-1" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148682 2577 flags.go:64] FLAG: --pods-per-core="0" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148685 2577 flags.go:64] FLAG: --port="10250" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148689 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148692 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0da87e7e4c9066ffd" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148695 2577 flags.go:64] FLAG: --qos-reserved="" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148698 2577 flags.go:64] FLAG: --read-only-port="10255" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148701 2577 flags.go:64] FLAG: --register-node="true" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148704 2577 flags.go:64] FLAG: --register-with-taints="" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148708 2577 flags.go:64] FLAG: --registry-burst="10" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148711 2577 flags.go:64] FLAG: --registry-qps="5" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148714 2577 flags.go:64] FLAG: --reserved-cpus="" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148717 2577 flags.go:64] FLAG: --reserved-memory="" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148721 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148724 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148727 2577 flags.go:64] FLAG: --rotate-certificates="false" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148730 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148733 2577 flags.go:64] FLAG: --runonce="false" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148735 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 09 02:30:12.155406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148740 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148743 2577 flags.go:64] FLAG: --seccomp-default="false" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148746 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148749 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148752 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148755 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148758 2577 flags.go:64] FLAG: --storage-driver-password="root" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148761 2577 flags.go:64] FLAG: --storage-driver-secure="false" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148764 2577 flags.go:64] FLAG: --storage-driver-table="stats" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148767 2577 flags.go:64] FLAG: --storage-driver-user="root" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148770 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148773 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148776 2577 flags.go:64] FLAG: --system-cgroups="" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148779 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148786 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148789 2577 flags.go:64] FLAG: --tls-cert-file="" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148792 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148797 2577 flags.go:64] FLAG: --tls-min-version="" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148800 2577 flags.go:64] FLAG: --tls-private-key-file="" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148802 2577 flags.go:64] FLAG: --topology-manager-policy="none" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148805 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148809 2577 flags.go:64] FLAG: --topology-manager-scope="container" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148812 2577 flags.go:64] FLAG: --v="2" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148831 2577 flags.go:64] FLAG: --version="false" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148839 2577 flags.go:64] FLAG: --vmodule="" Sep 09 02:30:12.156118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148844 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 09 02:30:12.156710 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.148848 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 09 02:30:12.156710 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.148972 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 02:30:12.156710 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.148976 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 02:30:12.156710 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.148979 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 02:30:12.156710 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.148981 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 02:30:12.156710 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.148984 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 09 02:30:12.156710 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.148988 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 02:30:12.156710 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.148990 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 02:30:12.156710 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.148993 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 02:30:12.156710 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.148995 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 02:30:12.156710 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.148998 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 02:30:12.156710 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149001 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 02:30:12.156710 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149004 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 02:30:12.156710 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149006 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 02:30:12.156710 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149009 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 02:30:12.156710 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149012 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 02:30:12.156710 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149014 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 02:30:12.156710 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149017 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 02:30:12.156710 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149019 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 02:30:12.156710 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149022 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 02:30:12.157222 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149025 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 02:30:12.157222 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149027 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 02:30:12.157222 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149030 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 02:30:12.157222 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149033 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 02:30:12.157222 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149035 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 02:30:12.157222 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149038 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 02:30:12.157222 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149041 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 02:30:12.157222 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149043 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 02:30:12.157222 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149046 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 02:30:12.157222 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149048 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 02:30:12.157222 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149051 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 02:30:12.157222 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149054 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 02:30:12.157222 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149057 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 02:30:12.157222 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149059 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 02:30:12.157222 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149064 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 02:30:12.157222 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149068 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 02:30:12.157222 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149071 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 02:30:12.157222 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149074 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 02:30:12.157222 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149076 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 02:30:12.157691 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149079 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 02:30:12.157691 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149082 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 02:30:12.157691 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149085 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 02:30:12.157691 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149089 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 02:30:12.157691 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149091 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 02:30:12.157691 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149094 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 02:30:12.157691 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149097 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 02:30:12.157691 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149101 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 02:30:12.157691 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149104 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 02:30:12.157691 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149107 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 02:30:12.157691 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149109 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 02:30:12.157691 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149112 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 02:30:12.157691 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149114 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 02:30:12.157691 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149117 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 02:30:12.157691 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149120 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 02:30:12.157691 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149122 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 02:30:12.157691 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149125 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 02:30:12.157691 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149127 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 02:30:12.157691 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149130 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 02:30:12.158175 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149132 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 02:30:12.158175 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149135 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 02:30:12.158175 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149137 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 02:30:12.158175 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149140 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 02:30:12.158175 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149142 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 02:30:12.158175 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149145 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 02:30:12.158175 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149148 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 02:30:12.158175 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149150 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 02:30:12.158175 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149153 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 02:30:12.158175 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149155 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 02:30:12.158175 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149158 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 02:30:12.158175 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149161 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 02:30:12.158175 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149164 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 02:30:12.158175 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149166 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 02:30:12.158175 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149169 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 02:30:12.158175 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149171 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 02:30:12.158175 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149174 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 02:30:12.158175 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149177 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 02:30:12.158175 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149179 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 02:30:12.158175 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149182 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 02:30:12.158659 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149186 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 02:30:12.158659 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149188 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 02:30:12.158659 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149192 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 02:30:12.158659 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149196 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 02:30:12.158659 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149199 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 02:30:12.158659 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149203 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 02:30:12.158659 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149206 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 02:30:12.158659 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149209 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 02:30:12.158659 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149212 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 02:30:12.158659 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149214 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 02:30:12.158659 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149217 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 02:30:12.158659 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149220 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 02:30:12.158659 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149222 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 02:30:12.158659 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149225 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 02:30:12.158659 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149228 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 02:30:12.158659 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149231 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 02:30:12.158659 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149233 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 02:30:12.158659 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149236 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 02:30:12.158659 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149238 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 02:30:12.159193 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149241 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 02:30:12.159193 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149244 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 02:30:12.159193 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.149247 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 02:30:12.159193 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.149895 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 02:30:12.159193 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.157396 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 09 02:30:12.159193 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.157413 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 02:30:12.159193 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.157432 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 09 02:30:12.159193 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.157442 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 09 02:30:12.159193 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157513 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 02:30:12.159193 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157518 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 02:30:12.159193 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157522 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 02:30:12.159193 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157525 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 02:30:12.159193 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157529 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 02:30:12.159193 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157533 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 02:30:12.159548 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157538 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 02:30:12.159548 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157541 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 02:30:12.159548 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157544 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 02:30:12.159548 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157547 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 02:30:12.159548 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157549 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 02:30:12.159548 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157552 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 02:30:12.159548 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157555 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 02:30:12.159548 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157557 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 02:30:12.159548 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157560 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 02:30:12.159548 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157563 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 02:30:12.159548 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157565 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 02:30:12.159548 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157568 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 02:30:12.159548 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157571 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 02:30:12.159548 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157573 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 02:30:12.159548 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157576 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 02:30:12.159548 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157579 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 02:30:12.159548 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157581 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 02:30:12.159548 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157584 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 02:30:12.159548 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157588 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 02:30:12.159548 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157590 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 02:30:12.160117 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157593 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 02:30:12.160117 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157596 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 02:30:12.160117 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157598 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 02:30:12.160117 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157602 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 02:30:12.160117 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157605 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 02:30:12.160117 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157608 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 02:30:12.160117 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157610 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 02:30:12.160117 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157613 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 02:30:12.160117 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157616 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 02:30:12.160117 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157618 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 02:30:12.160117 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157621 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 02:30:12.160117 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157623 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 02:30:12.160117 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157626 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 02:30:12.160117 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157629 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 02:30:12.160117 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157631 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 02:30:12.160117 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157634 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 02:30:12.160117 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157636 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 02:30:12.160117 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157639 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 02:30:12.160117 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157642 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 02:30:12.160580 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157644 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 02:30:12.160580 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157647 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 02:30:12.160580 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157650 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 02:30:12.160580 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157652 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 02:30:12.160580 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157655 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 02:30:12.160580 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157657 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 02:30:12.160580 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157660 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 02:30:12.160580 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157663 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 02:30:12.160580 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157665 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 02:30:12.160580 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157668 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 02:30:12.160580 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157670 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 02:30:12.160580 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157673 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 02:30:12.160580 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157676 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 02:30:12.160580 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157679 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 02:30:12.160580 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157682 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 02:30:12.160580 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157684 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 02:30:12.160580 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157687 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 02:30:12.160580 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157690 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 02:30:12.160580 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157693 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 02:30:12.160580 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157698 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 02:30:12.161081 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157701 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 02:30:12.161081 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157704 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 02:30:12.161081 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157707 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 02:30:12.161081 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157710 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 02:30:12.161081 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157712 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 02:30:12.161081 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157715 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 02:30:12.161081 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157717 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 02:30:12.161081 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157720 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 02:30:12.161081 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157724 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 02:30:12.161081 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157727 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 02:30:12.161081 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157730 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 02:30:12.161081 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157733 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 02:30:12.161081 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157736 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 02:30:12.161081 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157739 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 02:30:12.161081 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157742 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 09 02:30:12.161081 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157744 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 02:30:12.161081 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157747 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 02:30:12.161081 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157750 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 02:30:12.161081 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157752 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 02:30:12.161081 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157755 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 02:30:12.161562 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157758 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 02:30:12.161562 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157760 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 02:30:12.161562 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157762 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 02:30:12.161562 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157765 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 02:30:12.161562 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157767 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 02:30:12.161562 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157773 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 02:30:12.161562 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157776 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 02:30:12.161562 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157779 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 02:30:12.161562 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157781 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 02:30:12.161562 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157784 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 02:30:12.161562 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157787 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 02:30:12.161562 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157789 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 02:30:12.161562 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157792 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 02:30:12.161562 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157795 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 02:30:12.161562 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.157800 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 02:30:12.161950 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157939 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 02:30:12.161950 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157945 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 02:30:12.161950 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157949 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 02:30:12.161950 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157952 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 02:30:12.161950 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157955 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 02:30:12.161950 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157959 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 02:30:12.161950 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157962 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 02:30:12.161950 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157965 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 02:30:12.161950 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157967 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 02:30:12.161950 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157970 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 02:30:12.161950 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157973 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 02:30:12.161950 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157975 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 02:30:12.161950 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157978 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 02:30:12.161950 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157981 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 02:30:12.161950 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157983 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 02:30:12.161950 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157986 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 02:30:12.161950 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157989 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 02:30:12.161950 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157991 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 02:30:12.161950 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157994 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 02:30:12.162397 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157996 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 02:30:12.162397 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.157999 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 02:30:12.162397 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158001 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 02:30:12.162397 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158004 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 02:30:12.162397 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158007 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 02:30:12.162397 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158010 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 02:30:12.162397 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158012 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 02:30:12.162397 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158015 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 02:30:12.162397 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158018 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 02:30:12.162397 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158021 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 02:30:12.162397 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158023 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 02:30:12.162397 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158026 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 02:30:12.162397 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158028 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 02:30:12.162397 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158031 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 02:30:12.162397 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158034 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 02:30:12.162397 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158037 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 02:30:12.162397 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158040 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 02:30:12.162397 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158042 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 02:30:12.162397 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158045 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 02:30:12.162397 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158048 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 02:30:12.162902 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158051 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 02:30:12.162902 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158053 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 02:30:12.162902 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158056 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 02:30:12.162902 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158059 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 02:30:12.162902 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158061 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 02:30:12.162902 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158064 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 02:30:12.162902 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158066 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 02:30:12.162902 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158069 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 02:30:12.162902 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158071 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 02:30:12.162902 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158074 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 02:30:12.162902 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158077 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 02:30:12.162902 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158081 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 02:30:12.162902 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158084 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 02:30:12.162902 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158087 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 02:30:12.162902 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158090 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 02:30:12.162902 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158092 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 02:30:12.162902 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158095 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 02:30:12.162902 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158099 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 02:30:12.162902 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158103 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 02:30:12.163366 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158106 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 02:30:12.163366 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158109 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 02:30:12.163366 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158112 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 02:30:12.163366 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158114 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 02:30:12.163366 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158117 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 02:30:12.163366 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158120 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 02:30:12.163366 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158122 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 02:30:12.163366 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158125 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 02:30:12.163366 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158128 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 02:30:12.163366 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158130 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 02:30:12.163366 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158133 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 02:30:12.163366 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158135 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 02:30:12.163366 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158138 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 02:30:12.163366 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158140 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 02:30:12.163366 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158143 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 02:30:12.163366 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158146 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 02:30:12.163366 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158148 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 02:30:12.163366 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158150 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 02:30:12.163366 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158153 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 02:30:12.163366 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158155 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 02:30:12.163913 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158158 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 02:30:12.163913 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158160 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 02:30:12.163913 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158163 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 02:30:12.163913 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158165 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 02:30:12.163913 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158168 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 02:30:12.163913 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158170 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 09 02:30:12.163913 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158173 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 02:30:12.163913 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158176 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 02:30:12.163913 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158179 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 02:30:12.163913 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158181 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 02:30:12.163913 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158184 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 02:30:12.163913 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158187 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 02:30:12.163913 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158190 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 02:30:12.163913 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158193 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 02:30:12.163913 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158195 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 02:30:12.163913 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158198 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 02:30:12.163913 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158200 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 02:30:12.163913 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158203 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 02:30:12.163913 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158205 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 02:30:12.163913 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158208 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 02:30:12.164414 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:12.158211 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 02:30:12.164414 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.158215 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 02:30:12.164414 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.158331 2577 server.go:962] "Client rotation is on, will bootstrap in background" Sep 09 02:30:12.164414 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.160531 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 09 02:30:12.164414 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.161774 2577 server.go:1019] "Starting client certificate rotation" Sep 09 02:30:12.164414 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.161850 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 02:30:12.164414 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.161891 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 02:30:12.164414 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.164303 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 02:30:12.165794 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.165401 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 02:30:12.176527 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.176406 2577 log.go:25] "Validated CRI v1 runtime API" Sep 09 02:30:12.180790 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.180770 2577 log.go:25] "Validated CRI v1 image API" Sep 09 02:30:12.181975 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.181869 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 09 02:30:12.191931 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.191910 2577 fs.go:135] Filesystem UUIDs: map[27e272cd-9807-4356-b652-f27e13b9069d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 af3a0bc5-d416-4e2d-827c-c56d80d6267c:/dev/nvme0n1p3] Sep 09 02:30:12.191994 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.191929 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 09 02:30:12.193964 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.193948 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 02:30:12.198409 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.198297 2577 manager.go:216] Machine: {Timestamp:2026-09-09 02:30:12.197060196 +0000 UTC m=+0.327302283 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099278 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f3ea8f76ad57ac504667b3b4d22ca SystemUUID:ec2f3ea8-f76a-d57a-c504-667b3b4d22ca BootID:a2bc7d4a-082e-4e8c-af8a-1b2290ba4556 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d2:1f:fb:7e:c3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d2:1f:fb:7e:c3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:3b:6b:21:9b:96 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 09 02:30:12.198409 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.198404 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 09 02:30:12.198538 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.198526 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 09 02:30:12.200088 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.200050 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 09 02:30:12.200250 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.200091 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-37.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 09 02:30:12.200777 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.200766 2577 topology_manager.go:138] "Creating topology manager with none policy" Sep 09 02:30:12.200850 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.200784 2577 container_manager_linux.go:309] "Creating device plugin manager" Sep 09 02:30:12.200850 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.200795 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 02:30:12.201760 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.201748 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 02:30:12.203445 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.203433 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 09 02:30:12.207710 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.207697 2577 state_mem.go:36] "Initialized new in-memory state store" Sep 09 02:30:12.207859 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.207848 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 09 02:30:12.207915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.207888 2577 kubelet.go:486] "Attempting to sync node with API server" Sep 09 02:30:12.207915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.207902 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 09 02:30:12.207998 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.207922 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 09 02:30:12.207998 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.207936 2577 kubelet.go:398] "Adding apiserver pod source" Sep 09 02:30:12.207998 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.207948 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 09 02:30:12.208352 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.208338 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 02:30:12.208419 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.208361 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 02:30:12.211560 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.211539 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 09 02:30:12.214673 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.214659 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 09 02:30:12.214728 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.214690 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 09 02:30:12.216123 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.216112 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 09 02:30:12.216156 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.216128 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 09 02:30:12.216156 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.216136 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 09 02:30:12.216156 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.216143 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 09 02:30:12.216156 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.216156 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 09 02:30:12.216267 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.216162 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 09 02:30:12.216267 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.216175 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 09 02:30:12.216267 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.216180 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 09 02:30:12.216267 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.216188 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 09 02:30:12.216267 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.216194 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 09 02:30:12.216267 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.216202 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 09 02:30:12.216267 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.216207 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 09 02:30:12.217810 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.217801 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 09 02:30:12.217860 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.217830 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 09 02:30:12.221943 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.221926 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-37.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 09 02:30:12.222034 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.221939 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 09 02:30:12.222125 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.222103 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-37.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 09 02:30:12.223186 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.223173 2577 server.go:1268] "Started kubelet" Sep 09 02:30:12.223256 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.223230 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 09 02:30:12.223330 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.223305 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 09 02:30:12.223377 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.223350 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 09 02:30:12.224237 ip-10-0-137-37 systemd[1]: Started Kubernetes Kubelet. Sep 09 02:30:12.225041 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.225012 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 09 02:30:12.226618 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.226600 2577 server.go:310] "Adding debug handlers to kubelet server" Sep 09 02:30:12.229149 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.228975 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tj9nj" Sep 09 02:30:12.229832 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.228923 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-37.ec2.internal.18d3862eb8cae850 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-37.ec2.internal,UID:ip-10-0-137-37.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-37.ec2.internal,},FirstTimestamp:2026-09-09 02:30:12.223141968 +0000 UTC m=+0.353384056,LastTimestamp:2026-09-09 02:30:12.223141968 +0000 UTC m=+0.353384056,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-37.ec2.internal,}" Sep 09 02:30:12.229946 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.229811 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 09 02:30:12.232445 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.232427 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 09 02:30:12.232445 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.232442 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 09 02:30:12.232669 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.232648 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-37.ec2.internal\" not found" Sep 09 02:30:12.233171 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.233154 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 09 02:30:12.233171 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.233156 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 09 02:30:12.233307 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.233182 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 09 02:30:12.233307 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.233230 2577 reconstruct.go:97] "Volume reconstruction finished" Sep 09 02:30:12.233307 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.233240 2577 reconciler.go:29] "Reconciler: start to sync state" Sep 09 02:30:12.233307 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.233288 2577 factory.go:55] Registering systemd factory Sep 09 02:30:12.233307 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.233305 2577 factory.go:223] Registration of the systemd container factory successfully Sep 09 02:30:12.233503 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.233485 2577 factory.go:153] Registering CRI-O factory Sep 09 02:30:12.233550 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.233507 2577 factory.go:223] Registration of the crio container factory successfully Sep 09 02:30:12.233604 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.233565 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 09 02:30:12.233604 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.233584 2577 factory.go:103] Registering Raw factory Sep 09 02:30:12.233604 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.233601 2577 manager.go:1196] Started watching for new ooms in manager Sep 09 02:30:12.233949 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.233937 2577 manager.go:349] Starting recovery of all containers Sep 09 02:30:12.236455 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.236431 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 09 02:30:12.236558 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.236523 2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-37.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 09 02:30:12.239972 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.239946 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tj9nj" Sep 09 02:30:12.242322 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.242135 2577 manager.go:354] Recovery completed Sep 09 02:30:12.246723 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.246708 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 02:30:12.248541 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.248525 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-37.ec2.internal" event="NodeHasSufficientMemory" Sep 09 02:30:12.248630 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.248558 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-37.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 02:30:12.248630 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.248571 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-37.ec2.internal" event="NodeHasSufficientPID" Sep 09 02:30:12.249256 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.249198 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 09 02:30:12.249256 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.249208 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 09 02:30:12.249256 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.249223 2577 state_mem.go:36] "Initialized new in-memory state store" Sep 09 02:30:12.249256 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.249226 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 02:30:12.250504 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.250491 2577 policy_none.go:49] "None policy: Start" Sep 09 02:30:12.250554 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.250511 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 09 02:30:12.250554 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.250525 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 09 02:30:12.250924 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.250905 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-37.ec2.internal" event="NodeHasSufficientMemory" Sep 09 02:30:12.251022 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.250938 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-37.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 02:30:12.251022 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.250953 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-37.ec2.internal" event="NodeHasSufficientPID" Sep 09 02:30:12.301919 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.301891 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 09 02:30:12.396594 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.303021 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 09 02:30:12.396594 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.303083 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 09 02:30:12.396594 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.303112 2577 kubelet.go:2448] "Starting kubelet main sync loop" Sep 09 02:30:12.396594 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.303159 2577 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Sep 09 02:30:12.396594 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.307578 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 02:30:12.396594 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.333641 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-37.ec2.internal\" not found" Sep 09 02:30:12.396594 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.396050 2577 policy_none.go:47] "Start" Sep 09 02:30:12.403756 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.403736 2577 kubelet.go:2472] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Sep 09 02:30:12.434292 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.434262 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-37.ec2.internal\" not found" Sep 09 02:30:12.437395 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.437376 2577 manager.go:337] "Starting Device Plugin manager" Sep 09 02:30:12.437502 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.437428 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 09 02:30:12.437502 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.437443 2577 server.go:87] "Starting device plugin registration server" Sep 09 02:30:12.437788 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.437776 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 09 02:30:12.437854 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.437789 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 09 02:30:12.437915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.437897 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 09 02:30:12.437991 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.437979 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 09 02:30:12.438043 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.437992 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 09 02:30:12.438555 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.438536 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 09 02:30:12.438640 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.438580 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-37.ec2.internal\" not found" Sep 09 02:30:12.447402 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.447387 2577 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-37.ec2.internal\" not found" node="ip-10-0-137-37.ec2.internal" Sep 09 02:30:12.538861 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.538839 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 02:30:12.539845 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.539830 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-37.ec2.internal" event="NodeHasSufficientMemory" Sep 09 02:30:12.539902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.539858 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-37.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 02:30:12.539902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.539869 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-37.ec2.internal" event="NodeHasSufficientPID" Sep 09 02:30:12.539902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.539898 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-37.ec2.internal" Sep 09 02:30:12.549956 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.549940 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-37.ec2.internal" Sep 09 02:30:12.550008 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.549961 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-37.ec2.internal\": node \"ip-10-0-137-37.ec2.internal\" not found" Sep 09 02:30:12.569758 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.569738 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-37.ec2.internal\" not found" Sep 09 02:30:12.604878 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.604838 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-37.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-37.ec2.internal"] Sep 09 02:30:12.605004 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.604942 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 02:30:12.606534 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.606518 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-37.ec2.internal" event="NodeHasSufficientMemory" Sep 09 02:30:12.606605 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.606551 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-37.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 02:30:12.606605 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.606563 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-37.ec2.internal" event="NodeHasSufficientPID" Sep 09 02:30:12.607880 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.607867 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 02:30:12.608003 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.607989 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-37.ec2.internal" Sep 09 02:30:12.608073 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.608028 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 02:30:12.608599 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.608582 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-37.ec2.internal" event="NodeHasSufficientMemory" Sep 09 02:30:12.608648 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.608610 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-37.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 02:30:12.608648 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.608620 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-37.ec2.internal" event="NodeHasSufficientPID" Sep 09 02:30:12.608707 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.608582 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-37.ec2.internal" event="NodeHasSufficientMemory" Sep 09 02:30:12.608707 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.608693 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-37.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 02:30:12.608707 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.608705 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-37.ec2.internal" event="NodeHasSufficientPID" Sep 09 02:30:12.609793 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.609778 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-37.ec2.internal" Sep 09 02:30:12.609860 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.609808 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 02:30:12.610533 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.610517 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-37.ec2.internal" event="NodeHasSufficientMemory" Sep 09 02:30:12.610590 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.610549 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-37.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 02:30:12.610590 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.610565 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-37.ec2.internal" event="NodeHasSufficientPID" Sep 09 02:30:12.646317 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.646293 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-37.ec2.internal\" not found" node="ip-10-0-137-37.ec2.internal" Sep 09 02:30:12.650437 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.650422 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-37.ec2.internal\" not found" node="ip-10-0-137-37.ec2.internal" Sep 09 02:30:12.670616 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.670598 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-37.ec2.internal\" not found" Sep 09 02:30:12.736003 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.735978 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7a47394d5f75f79888655dda3114eff2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-37.ec2.internal\" (UID: \"7a47394d5f75f79888655dda3114eff2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-37.ec2.internal" Sep 09 02:30:12.736003 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.736004 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a47394d5f75f79888655dda3114eff2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-37.ec2.internal\" (UID: \"7a47394d5f75f79888655dda3114eff2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-37.ec2.internal" Sep 09 02:30:12.736130 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.736020 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/60f11ba69470c54c65a13a194197ca4c-config\") pod \"kube-apiserver-proxy-ip-10-0-137-37.ec2.internal\" (UID: \"60f11ba69470c54c65a13a194197ca4c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-37.ec2.internal" Sep 09 02:30:12.771386 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.771369 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-37.ec2.internal\" not found" Sep 09 02:30:12.836391 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.836371 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7a47394d5f75f79888655dda3114eff2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-37.ec2.internal\" (UID: \"7a47394d5f75f79888655dda3114eff2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-37.ec2.internal" Sep 09 02:30:12.836448 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.836396 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a47394d5f75f79888655dda3114eff2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-37.ec2.internal\" (UID: \"7a47394d5f75f79888655dda3114eff2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-37.ec2.internal" Sep 09 02:30:12.836448 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.836412 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/60f11ba69470c54c65a13a194197ca4c-config\") pod \"kube-apiserver-proxy-ip-10-0-137-37.ec2.internal\" (UID: \"60f11ba69470c54c65a13a194197ca4c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-37.ec2.internal" Sep 09 02:30:12.836543 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.836449 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/60f11ba69470c54c65a13a194197ca4c-config\") pod \"kube-apiserver-proxy-ip-10-0-137-37.ec2.internal\" (UID: \"60f11ba69470c54c65a13a194197ca4c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-37.ec2.internal" Sep 09 02:30:12.836543 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.836472 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7a47394d5f75f79888655dda3114eff2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-37.ec2.internal\" (UID: \"7a47394d5f75f79888655dda3114eff2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-37.ec2.internal" Sep 09 02:30:12.836543 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.836491 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a47394d5f75f79888655dda3114eff2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-37.ec2.internal\" (UID: \"7a47394d5f75f79888655dda3114eff2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-37.ec2.internal" Sep 09 02:30:12.871430 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.871407 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-37.ec2.internal\" not found" Sep 09 02:30:12.948910 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.948849 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-37.ec2.internal" Sep 09 02:30:12.954871 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:12.954854 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-37.ec2.internal" Sep 09 02:30:12.971779 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:12.971761 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-37.ec2.internal\" not found" Sep 09 02:30:13.072324 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:13.072294 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-37.ec2.internal\" not found" Sep 09 02:30:13.161738 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.161710 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 09 02:30:13.162134 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.161874 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 02:30:13.173066 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:13.173048 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-37.ec2.internal\" not found" Sep 09 02:30:13.184009 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.183991 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 02:30:13.208356 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.208302 2577 apiserver.go:52] "Watching apiserver" Sep 09 02:30:13.219326 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.219307 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 09 02:30:13.219649 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.219632 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-wlxqx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd","openshift-cluster-node-tuning-operator/tuned-6mzm8","openshift-multus/multus-9pqdb","openshift-ovn-kubernetes/ovnkube-node-t8g9x","kube-system/global-pull-secret-syncer-4fb84","kube-system/konnectivity-agent-9jpwt","openshift-dns/node-resolver-wr95f","openshift-image-registry/node-ca-qfvvm","openshift-multus/multus-additional-cni-plugins-2ddl4","openshift-multus/network-metrics-daemon-n7lh6","openshift-network-diagnostics/network-check-target-zr2ql"] Sep 09 02:30:13.221089 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.221075 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wlxqx" Sep 09 02:30:13.222328 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.222309 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.223515 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.223500 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.223937 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.223921 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c2dc5\"" Sep 09 02:30:13.225731 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.224538 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 09 02:30:13.225731 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.224591 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.225731 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.225209 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 09 02:30:13.225731 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.225261 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 09 02:30:13.225731 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.225269 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 09 02:30:13.225731 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.225534 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 09 02:30:13.226061 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.225804 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 09 02:30:13.226109 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.226085 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zr6gb\"" Sep 09 02:30:13.226353 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.226334 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6b4bg\"" Sep 09 02:30:13.226407 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.226381 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 09 02:30:13.226575 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.226545 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 09 02:30:13.227009 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.226993 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.227103 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.227027 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 09 02:30:13.227162 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.227145 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hxskh\"" Sep 09 02:30:13.227326 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.227306 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 09 02:30:13.228139 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.228121 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 09 02:30:13.228228 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.228142 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 09 02:30:13.228681 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.228663 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:13.228799 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:13.228776 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fb84" podUID="e0fea0ae-0424-4574-b264-04ea4bcdea35" Sep 09 02:30:13.229420 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.229405 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 09 02:30:13.229563 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.229551 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 09 02:30:13.229732 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.229692 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 09 02:30:13.229878 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.229863 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 09 02:30:13.229941 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.229905 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vbnll\"" Sep 09 02:30:13.230013 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.229999 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9jpwt" Sep 09 02:30:13.231013 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.230999 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 09 02:30:13.231081 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.231016 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 09 02:30:13.231223 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.231211 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wr95f" Sep 09 02:30:13.232442 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.232427 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qfvvm" Sep 09 02:30:13.232575 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.232558 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 09 02:30:13.232743 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.232729 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-37.ec2.internal" Sep 09 02:30:13.232947 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.232927 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 09 02:30:13.233039 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.232986 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 09 02:30:13.234417 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.234090 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 09 02:30:13.234417 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.234101 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 09 02:30:13.234417 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.234128 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qhx7r\"" Sep 09 02:30:13.234417 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.234189 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.234417 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.234155 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sj7xn\"" Sep 09 02:30:13.235099 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.235077 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 09 02:30:13.235186 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.235100 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 09 02:30:13.235295 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.235281 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 09 02:30:13.235486 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.235466 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rpqjd\"" Sep 09 02:30:13.236405 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.236387 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:13.236525 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:13.236447 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7lh6" podUID="59a52c60-f713-4a2f-8de1-5d7a2d43ac41" Sep 09 02:30:13.236525 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.236466 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 09 02:30:13.236709 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.236584 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 09 02:30:13.236811 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.236791 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fs8xm\"" Sep 09 02:30:13.237725 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.237710 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-lib-modules\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.237787 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.237725 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:13.237787 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.237736 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-var-lib-openvswitch\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.237787 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.237754 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-host-cni-netd\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.237787 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.237771 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6022d651-c68e-42a6-86c9-eee48a15f8a2-konnectivity-ca\") pod \"konnectivity-agent-9jpwt\" (UID: \"6022d651-c68e-42a6-86c9-eee48a15f8a2\") " pod="kube-system/konnectivity-agent-9jpwt" Sep 09 02:30:13.237787 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.237785 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-host-run-k8s-cni-cncf-io\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.237968 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.237802 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-host-run-netns\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.237968 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:13.237781 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2ql" podUID="6e6ad2e4-55a9-4509-99b4-30958e61e3a6" Sep 09 02:30:13.237968 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.237856 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-multus-conf-dir\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.237968 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.237876 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-etc-sysctl-d\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.237968 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.237912 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-var-lib-kubelet\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.237968 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.237927 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-run-openvswitch\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.237968 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.237940 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63637b1b-65f1-4df2-b182-6a673b4f9d6d-ovnkube-config\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.237968 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.237956 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/326af4c7-6a8d-4510-92f6-36ba2efe0943-iptables-alerter-script\") pod \"iptables-alerter-wlxqx\" (UID: \"326af4c7-6a8d-4510-92f6-36ba2efe0943\") " pod="openshift-network-operator/iptables-alerter-wlxqx" Sep 09 02:30:13.237968 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.237970 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-system-cni-dir\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.238208 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.237985 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-host-var-lib-cni-bin\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.238208 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.237998 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-hostroot\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.238208 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238012 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-etc-modprobe-d\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.238208 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238025 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-etc-systemd\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.238208 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238039 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret\") pod \"global-pull-secret-syncer-4fb84\" (UID: \"e0fea0ae-0424-4574-b264-04ea4bcdea35\") " pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:13.238208 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238060 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b77648c-d60e-4ba3-806f-ea2f240ac764-cni-binary-copy\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.238208 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238080 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-tmp\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.238208 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238103 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-systemd-units\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.238208 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238118 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-etc-sysconfig\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.238208 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238141 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-sys\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.238208 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238158 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-host-kubelet\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.238208 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238183 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6022d651-c68e-42a6-86c9-eee48a15f8a2-agent-certs\") pod \"konnectivity-agent-9jpwt\" (UID: \"6022d651-c68e-42a6-86c9-eee48a15f8a2\") " pod="kube-system/konnectivity-agent-9jpwt" Sep 09 02:30:13.238659 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238211 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/326af4c7-6a8d-4510-92f6-36ba2efe0943-host-slash\") pod \"iptables-alerter-wlxqx\" (UID: \"326af4c7-6a8d-4510-92f6-36ba2efe0943\") " pod="openshift-network-operator/iptables-alerter-wlxqx" Sep 09 02:30:13.238659 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238232 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2cfd0ce1-cffe-43a7-832b-3e845b5d391d-registration-dir\") pod \"aws-ebs-csi-driver-node-s8gnd\" (UID: \"2cfd0ce1-cffe-43a7-832b-3e845b5d391d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.238659 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238245 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-host-slash\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.238659 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238259 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-etc-openvswitch\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.238659 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238275 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2f959dac-774e-473c-b318-7cec8464e980-serviceca\") pod \"node-ca-qfvvm\" (UID: \"2f959dac-774e-473c-b318-7cec8464e980\") " pod="openshift-image-registry/node-ca-qfvvm" Sep 09 02:30:13.238659 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238303 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vphzl\" (UniqueName: \"kubernetes.io/projected/326af4c7-6a8d-4510-92f6-36ba2efe0943-kube-api-access-vphzl\") pod \"iptables-alerter-wlxqx\" (UID: \"326af4c7-6a8d-4510-92f6-36ba2efe0943\") " pod="openshift-network-operator/iptables-alerter-wlxqx" Sep 09 02:30:13.238659 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238322 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2cfd0ce1-cffe-43a7-832b-3e845b5d391d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8gnd\" (UID: \"2cfd0ce1-cffe-43a7-832b-3e845b5d391d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.238659 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238340 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-log-socket\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.238659 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238369 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.238659 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238389 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0fea0ae-0424-4574-b264-04ea4bcdea35-kubelet-config\") pod \"global-pull-secret-syncer-4fb84\" (UID: \"e0fea0ae-0424-4574-b264-04ea4bcdea35\") " pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:13.238659 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238409 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-host-var-lib-kubelet\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.238659 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238439 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bddd2af-4d25-4c54-8f2d-800934a15ab2-cni-binary-copy\") pod \"multus-additional-cni-plugins-2ddl4\" (UID: \"7bddd2af-4d25-4c54-8f2d-800934a15ab2\") " pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.238659 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238477 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7bddd2af-4d25-4c54-8f2d-800934a15ab2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2ddl4\" (UID: \"7bddd2af-4d25-4c54-8f2d-800934a15ab2\") " pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.238659 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238511 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-host-run-netns\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.238659 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238534 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63637b1b-65f1-4df2-b182-6a673b4f9d6d-env-overrides\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.238659 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238562 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4b77648c-d60e-4ba3-806f-ea2f240ac764-multus-daemon-config\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.239419 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238589 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2d5c\" (UniqueName: \"kubernetes.io/projected/7bddd2af-4d25-4c54-8f2d-800934a15ab2-kube-api-access-d2d5c\") pod \"multus-additional-cni-plugins-2ddl4\" (UID: \"7bddd2af-4d25-4c54-8f2d-800934a15ab2\") " pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.239419 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238615 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-host-cni-bin\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.239419 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238637 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9wg8\" (UniqueName: \"kubernetes.io/projected/63637b1b-65f1-4df2-b182-6a673b4f9d6d-kube-api-access-b9wg8\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.239419 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238661 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-multus-cni-dir\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.239419 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238685 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-cnibin\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.239419 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238708 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-host-var-lib-cni-multus\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.239419 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238731 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-etc-sysctl-conf\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.239419 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238753 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-node-log\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.239419 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238775 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40631b35-00b4-4a4b-ab2e-f769434da1f1-hosts-file\") pod \"node-resolver-wr95f\" (UID: \"40631b35-00b4-4a4b-ab2e-f769434da1f1\") " pod="openshift-dns/node-resolver-wr95f" Sep 09 02:30:13.239419 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238795 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40631b35-00b4-4a4b-ab2e-f769434da1f1-tmp-dir\") pod \"node-resolver-wr95f\" (UID: \"40631b35-00b4-4a4b-ab2e-f769434da1f1\") " pod="openshift-dns/node-resolver-wr95f" Sep 09 02:30:13.239419 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238833 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2cfd0ce1-cffe-43a7-832b-3e845b5d391d-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8gnd\" (UID: \"2cfd0ce1-cffe-43a7-832b-3e845b5d391d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.239419 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238857 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-run\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.239419 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238887 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6jqc\" (UniqueName: \"kubernetes.io/projected/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-kube-api-access-s6jqc\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.239419 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238908 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-run-ovn\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.239419 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238933 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-host-run-ovn-kubernetes\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.239419 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238950 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-host-run-multus-certs\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.239419 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238971 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-etc-kubernetes\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.240263 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.238998 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-multus-socket-dir-parent\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.240263 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.239036 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7bddd2af-4d25-4c54-8f2d-800934a15ab2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2ddl4\" (UID: \"7bddd2af-4d25-4c54-8f2d-800934a15ab2\") " pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.240263 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.239070 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2cfd0ce1-cffe-43a7-832b-3e845b5d391d-socket-dir\") pod \"aws-ebs-csi-driver-node-s8gnd\" (UID: \"2cfd0ce1-cffe-43a7-832b-3e845b5d391d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.240263 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.239091 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2cfd0ce1-cffe-43a7-832b-3e845b5d391d-device-dir\") pod \"aws-ebs-csi-driver-node-s8gnd\" (UID: \"2cfd0ce1-cffe-43a7-832b-3e845b5d391d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.240263 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.239109 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2cfd0ce1-cffe-43a7-832b-3e845b5d391d-sys-fs\") pod \"aws-ebs-csi-driver-node-s8gnd\" (UID: \"2cfd0ce1-cffe-43a7-832b-3e845b5d391d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.240263 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.239123 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpwqr\" (UniqueName: \"kubernetes.io/projected/2cfd0ce1-cffe-43a7-832b-3e845b5d391d-kube-api-access-gpwqr\") pod \"aws-ebs-csi-driver-node-s8gnd\" (UID: \"2cfd0ce1-cffe-43a7-832b-3e845b5d391d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.240263 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.239154 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-host\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.240263 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.239179 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-etc-kubernetes\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.240263 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.239211 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bddd2af-4d25-4c54-8f2d-800934a15ab2-cnibin\") pod \"multus-additional-cni-plugins-2ddl4\" (UID: \"7bddd2af-4d25-4c54-8f2d-800934a15ab2\") " pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.240263 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.239238 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-run-systemd\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.240263 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.239256 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f959dac-774e-473c-b318-7cec8464e980-host\") pod \"node-ca-qfvvm\" (UID: \"2f959dac-774e-473c-b318-7cec8464e980\") " pod="openshift-image-registry/node-ca-qfvvm" Sep 09 02:30:13.240263 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.239273 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbpsx\" (UniqueName: \"kubernetes.io/projected/40631b35-00b4-4a4b-ab2e-f769434da1f1-kube-api-access-hbpsx\") pod \"node-resolver-wr95f\" (UID: \"40631b35-00b4-4a4b-ab2e-f769434da1f1\") " pod="openshift-dns/node-resolver-wr95f" Sep 09 02:30:13.240263 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.239295 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bddd2af-4d25-4c54-8f2d-800934a15ab2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2ddl4\" (UID: \"7bddd2af-4d25-4c54-8f2d-800934a15ab2\") " pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.240263 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.239321 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63637b1b-65f1-4df2-b182-6a673b4f9d6d-ovn-node-metrics-cert\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.240263 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.239344 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63637b1b-65f1-4df2-b182-6a673b4f9d6d-ovnkube-script-lib\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.240263 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.239363 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9bv8\" (UniqueName: \"kubernetes.io/projected/2f959dac-774e-473c-b318-7cec8464e980-kube-api-access-z9bv8\") pod \"node-ca-qfvvm\" (UID: \"2f959dac-774e-473c-b318-7cec8464e980\") " pod="openshift-image-registry/node-ca-qfvvm" Sep 09 02:30:13.241072 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.239377 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lzg2\" (UniqueName: \"kubernetes.io/projected/4b77648c-d60e-4ba3-806f-ea2f240ac764-kube-api-access-4lzg2\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.241072 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.239391 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bddd2af-4d25-4c54-8f2d-800934a15ab2-system-cni-dir\") pod \"multus-additional-cni-plugins-2ddl4\" (UID: \"7bddd2af-4d25-4c54-8f2d-800934a15ab2\") " pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.241072 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.239404 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-etc-tuned\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.241072 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.239416 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0fea0ae-0424-4574-b264-04ea4bcdea35-dbus\") pod \"global-pull-secret-syncer-4fb84\" (UID: \"e0fea0ae-0424-4574-b264-04ea4bcdea35\") " pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:13.244604 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.244587 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 02:30:13.244679 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.244656 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-37.ec2.internal" Sep 09 02:30:13.245111 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.245096 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-37.ec2.internal"] Sep 09 02:30:13.246773 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.246741 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 02:25:13 +0000 UTC" deadline="2028-02-03 05:16:33.70259099 +0000 UTC" Sep 09 02:30:13.246863 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.246773 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12290h46m20.455820013s" Sep 09 02:30:13.247888 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.247874 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 02:30:13.257342 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.257326 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 02:30:13.257833 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.257819 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-37.ec2.internal"] Sep 09 02:30:13.273745 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.273724 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wb4zr" Sep 09 02:30:13.284255 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.284237 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wb4zr" Sep 09 02:30:13.304153 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.304135 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 02:30:13.334487 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.334469 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 09 02:30:13.339599 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.339576 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-etc-sysconfig\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.339701 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.339616 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-sys\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.339701 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.339645 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-host-kubelet\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.339701 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.339675 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6022d651-c68e-42a6-86c9-eee48a15f8a2-agent-certs\") pod \"konnectivity-agent-9jpwt\" (UID: \"6022d651-c68e-42a6-86c9-eee48a15f8a2\") " pod="kube-system/konnectivity-agent-9jpwt" Sep 09 02:30:13.339701 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.339692 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-etc-sysconfig\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.339933 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.339700 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/326af4c7-6a8d-4510-92f6-36ba2efe0943-host-slash\") pod \"iptables-alerter-wlxqx\" (UID: \"326af4c7-6a8d-4510-92f6-36ba2efe0943\") " pod="openshift-network-operator/iptables-alerter-wlxqx" Sep 09 02:30:13.339933 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.339730 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-sys\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.339933 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.339750 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/326af4c7-6a8d-4510-92f6-36ba2efe0943-host-slash\") pod \"iptables-alerter-wlxqx\" (UID: \"326af4c7-6a8d-4510-92f6-36ba2efe0943\") " pod="openshift-network-operator/iptables-alerter-wlxqx" Sep 09 02:30:13.339933 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.339750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2cfd0ce1-cffe-43a7-832b-3e845b5d391d-registration-dir\") pod \"aws-ebs-csi-driver-node-s8gnd\" (UID: \"2cfd0ce1-cffe-43a7-832b-3e845b5d391d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.339933 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.339800 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-host-slash\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.339933 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.339805 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2cfd0ce1-cffe-43a7-832b-3e845b5d391d-registration-dir\") pod \"aws-ebs-csi-driver-node-s8gnd\" (UID: \"2cfd0ce1-cffe-43a7-832b-3e845b5d391d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.339933 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.339747 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-host-kubelet\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.339933 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.339854 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-host-slash\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.339933 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.339904 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-etc-openvswitch\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.340435 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.339954 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-etc-openvswitch\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.340435 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.339992 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2f959dac-774e-473c-b318-7cec8464e980-serviceca\") pod \"node-ca-qfvvm\" (UID: \"2f959dac-774e-473c-b318-7cec8464e980\") " pod="openshift-image-registry/node-ca-qfvvm" Sep 09 02:30:13.340435 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340018 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vphzl\" (UniqueName: \"kubernetes.io/projected/326af4c7-6a8d-4510-92f6-36ba2efe0943-kube-api-access-vphzl\") pod \"iptables-alerter-wlxqx\" (UID: \"326af4c7-6a8d-4510-92f6-36ba2efe0943\") " pod="openshift-network-operator/iptables-alerter-wlxqx" Sep 09 02:30:13.340435 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340044 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2cfd0ce1-cffe-43a7-832b-3e845b5d391d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8gnd\" (UID: \"2cfd0ce1-cffe-43a7-832b-3e845b5d391d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.340435 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340071 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-log-socket\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.340435 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340136 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-log-socket\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.340435 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340204 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2cfd0ce1-cffe-43a7-832b-3e845b5d391d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8gnd\" (UID: \"2cfd0ce1-cffe-43a7-832b-3e845b5d391d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.340435 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340346 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.340435 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340380 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0fea0ae-0424-4574-b264-04ea4bcdea35-kubelet-config\") pod \"global-pull-secret-syncer-4fb84\" (UID: \"e0fea0ae-0424-4574-b264-04ea4bcdea35\") " pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:13.340435 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340406 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-host-var-lib-kubelet\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.340435 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340430 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bddd2af-4d25-4c54-8f2d-800934a15ab2-cni-binary-copy\") pod \"multus-additional-cni-plugins-2ddl4\" (UID: \"7bddd2af-4d25-4c54-8f2d-800934a15ab2\") " pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.341014 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340452 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.341014 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340463 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 09 02:30:13.341014 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340515 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0fea0ae-0424-4574-b264-04ea4bcdea35-kubelet-config\") pod \"global-pull-secret-syncer-4fb84\" (UID: \"e0fea0ae-0424-4574-b264-04ea4bcdea35\") " pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:13.341014 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340561 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-host-var-lib-kubelet\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.341014 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340454 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7bddd2af-4d25-4c54-8f2d-800934a15ab2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2ddl4\" (UID: \"7bddd2af-4d25-4c54-8f2d-800934a15ab2\") " pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.341014 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340604 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjbwq\" (UniqueName: \"kubernetes.io/projected/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-kube-api-access-pjbwq\") pod \"network-metrics-daemon-n7lh6\" (UID: \"59a52c60-f713-4a2f-8de1-5d7a2d43ac41\") " pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:13.341014 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340632 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-host-run-netns\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.341014 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340657 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63637b1b-65f1-4df2-b182-6a673b4f9d6d-env-overrides\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.341014 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340684 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4b77648c-d60e-4ba3-806f-ea2f240ac764-multus-daemon-config\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.341014 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340690 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-host-run-netns\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.341014 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340711 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2d5c\" (UniqueName: \"kubernetes.io/projected/7bddd2af-4d25-4c54-8f2d-800934a15ab2-kube-api-access-d2d5c\") pod \"multus-additional-cni-plugins-2ddl4\" (UID: \"7bddd2af-4d25-4c54-8f2d-800934a15ab2\") " pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.341014 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340737 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-host-cni-bin\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.341014 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9wg8\" (UniqueName: \"kubernetes.io/projected/63637b1b-65f1-4df2-b182-6a673b4f9d6d-kube-api-access-b9wg8\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.341014 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340786 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-multus-cni-dir\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.341014 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340852 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-multus-cni-dir\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.341014 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340920 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-host-cni-bin\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.341014 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340958 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-cnibin\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.341014 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.340989 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-host-var-lib-cni-multus\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.341915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341042 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-etc-sysctl-conf\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.341915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341051 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-cnibin\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.341915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341067 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-node-log\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.341915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40631b35-00b4-4a4b-ab2e-f769434da1f1-hosts-file\") pod \"node-resolver-wr95f\" (UID: \"40631b35-00b4-4a4b-ab2e-f769434da1f1\") " pod="openshift-dns/node-resolver-wr95f" Sep 09 02:30:13.341915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341097 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-host-var-lib-cni-multus\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.341915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341134 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40631b35-00b4-4a4b-ab2e-f769434da1f1-tmp-dir\") pod \"node-resolver-wr95f\" (UID: \"40631b35-00b4-4a4b-ab2e-f769434da1f1\") " pod="openshift-dns/node-resolver-wr95f" Sep 09 02:30:13.341915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341157 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2cfd0ce1-cffe-43a7-832b-3e845b5d391d-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8gnd\" (UID: \"2cfd0ce1-cffe-43a7-832b-3e845b5d391d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.341915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341201 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-etc-sysctl-conf\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.341915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341230 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40631b35-00b4-4a4b-ab2e-f769434da1f1-hosts-file\") pod \"node-resolver-wr95f\" (UID: \"40631b35-00b4-4a4b-ab2e-f769434da1f1\") " pod="openshift-dns/node-resolver-wr95f" Sep 09 02:30:13.341915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341257 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-node-log\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.341915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341265 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-run\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.341915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341294 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6jqc\" (UniqueName: \"kubernetes.io/projected/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-kube-api-access-s6jqc\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.341915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341302 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2cfd0ce1-cffe-43a7-832b-3e845b5d391d-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8gnd\" (UID: \"2cfd0ce1-cffe-43a7-832b-3e845b5d391d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.341915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341321 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-run-ovn\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.341915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341348 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-host-run-ovn-kubernetes\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.341915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341381 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-host-run-multus-certs\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.341915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341401 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2f959dac-774e-473c-b318-7cec8464e980-serviceca\") pod \"node-ca-qfvvm\" (UID: \"2f959dac-774e-473c-b318-7cec8464e980\") " pod="openshift-image-registry/node-ca-qfvvm" Sep 09 02:30:13.341915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341410 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlt7n\" (UniqueName: \"kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n\") pod \"network-check-target-zr2ql\" (UID: \"6e6ad2e4-55a9-4509-99b4-30958e61e3a6\") " pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:13.342861 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341423 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bddd2af-4d25-4c54-8f2d-800934a15ab2-cni-binary-copy\") pod \"multus-additional-cni-plugins-2ddl4\" (UID: \"7bddd2af-4d25-4c54-8f2d-800934a15ab2\") " pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.342861 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341462 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-etc-kubernetes\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.342861 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341489 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-run\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.342861 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341491 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-multus-socket-dir-parent\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.342861 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341535 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7bddd2af-4d25-4c54-8f2d-800934a15ab2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2ddl4\" (UID: \"7bddd2af-4d25-4c54-8f2d-800934a15ab2\") " pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.342861 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341545 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-multus-socket-dir-parent\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.342861 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341566 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2cfd0ce1-cffe-43a7-832b-3e845b5d391d-socket-dir\") pod \"aws-ebs-csi-driver-node-s8gnd\" (UID: \"2cfd0ce1-cffe-43a7-832b-3e845b5d391d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.342861 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341588 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-host-run-ovn-kubernetes\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.342861 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341608 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2cfd0ce1-cffe-43a7-832b-3e845b5d391d-device-dir\") pod \"aws-ebs-csi-driver-node-s8gnd\" (UID: \"2cfd0ce1-cffe-43a7-832b-3e845b5d391d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.342861 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341636 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-run-ovn\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.342861 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341637 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40631b35-00b4-4a4b-ab2e-f769434da1f1-tmp-dir\") pod \"node-resolver-wr95f\" (UID: \"40631b35-00b4-4a4b-ab2e-f769434da1f1\") " pod="openshift-dns/node-resolver-wr95f" Sep 09 02:30:13.342861 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341658 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2cfd0ce1-cffe-43a7-832b-3e845b5d391d-socket-dir\") pod \"aws-ebs-csi-driver-node-s8gnd\" (UID: \"2cfd0ce1-cffe-43a7-832b-3e845b5d391d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.342861 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341662 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-host-run-multus-certs\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.342861 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2cfd0ce1-cffe-43a7-832b-3e845b5d391d-sys-fs\") pod \"aws-ebs-csi-driver-node-s8gnd\" (UID: \"2cfd0ce1-cffe-43a7-832b-3e845b5d391d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.342861 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341700 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-etc-kubernetes\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.342861 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341708 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2cfd0ce1-cffe-43a7-832b-3e845b5d391d-device-dir\") pod \"aws-ebs-csi-driver-node-s8gnd\" (UID: \"2cfd0ce1-cffe-43a7-832b-3e845b5d391d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.342861 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341710 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpwqr\" (UniqueName: \"kubernetes.io/projected/2cfd0ce1-cffe-43a7-832b-3e845b5d391d-kube-api-access-gpwqr\") pod \"aws-ebs-csi-driver-node-s8gnd\" (UID: \"2cfd0ce1-cffe-43a7-832b-3e845b5d391d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.343674 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341079 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7bddd2af-4d25-4c54-8f2d-800934a15ab2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2ddl4\" (UID: \"7bddd2af-4d25-4c54-8f2d-800934a15ab2\") " pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.343674 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341751 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2cfd0ce1-cffe-43a7-832b-3e845b5d391d-sys-fs\") pod \"aws-ebs-csi-driver-node-s8gnd\" (UID: \"2cfd0ce1-cffe-43a7-832b-3e845b5d391d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.343674 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341784 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-host\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.343674 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341853 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-etc-kubernetes\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.343674 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341870 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-host\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.343674 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341922 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bddd2af-4d25-4c54-8f2d-800934a15ab2-cnibin\") pod \"multus-additional-cni-plugins-2ddl4\" (UID: \"7bddd2af-4d25-4c54-8f2d-800934a15ab2\") " pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.343674 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341937 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-etc-kubernetes\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.343674 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341953 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-run-systemd\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.343674 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341980 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f959dac-774e-473c-b318-7cec8464e980-host\") pod \"node-ca-qfvvm\" (UID: \"2f959dac-774e-473c-b318-7cec8464e980\") " pod="openshift-image-registry/node-ca-qfvvm" Sep 09 02:30:13.343674 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.341995 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bddd2af-4d25-4c54-8f2d-800934a15ab2-cnibin\") pod \"multus-additional-cni-plugins-2ddl4\" (UID: \"7bddd2af-4d25-4c54-8f2d-800934a15ab2\") " pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.343674 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342008 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbpsx\" (UniqueName: \"kubernetes.io/projected/40631b35-00b4-4a4b-ab2e-f769434da1f1-kube-api-access-hbpsx\") pod \"node-resolver-wr95f\" (UID: \"40631b35-00b4-4a4b-ab2e-f769434da1f1\") " pod="openshift-dns/node-resolver-wr95f" Sep 09 02:30:13.343674 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342009 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-run-systemd\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.343674 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342048 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bddd2af-4d25-4c54-8f2d-800934a15ab2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2ddl4\" (UID: \"7bddd2af-4d25-4c54-8f2d-800934a15ab2\") " pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.343674 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342079 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs\") pod \"network-metrics-daemon-n7lh6\" (UID: \"59a52c60-f713-4a2f-8de1-5d7a2d43ac41\") " pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:13.343674 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342050 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f959dac-774e-473c-b318-7cec8464e980-host\") pod \"node-ca-qfvvm\" (UID: \"2f959dac-774e-473c-b318-7cec8464e980\") " pod="openshift-image-registry/node-ca-qfvvm" Sep 09 02:30:13.343674 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342109 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63637b1b-65f1-4df2-b182-6a673b4f9d6d-ovn-node-metrics-cert\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.343674 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342110 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63637b1b-65f1-4df2-b182-6a673b4f9d6d-env-overrides\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.344339 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342139 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7bddd2af-4d25-4c54-8f2d-800934a15ab2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2ddl4\" (UID: \"7bddd2af-4d25-4c54-8f2d-800934a15ab2\") " pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.344339 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342223 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bddd2af-4d25-4c54-8f2d-800934a15ab2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2ddl4\" (UID: \"7bddd2af-4d25-4c54-8f2d-800934a15ab2\") " pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.344339 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342226 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63637b1b-65f1-4df2-b182-6a673b4f9d6d-ovnkube-script-lib\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.344339 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342267 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9bv8\" (UniqueName: \"kubernetes.io/projected/2f959dac-774e-473c-b318-7cec8464e980-kube-api-access-z9bv8\") pod \"node-ca-qfvvm\" (UID: \"2f959dac-774e-473c-b318-7cec8464e980\") " pod="openshift-image-registry/node-ca-qfvvm" Sep 09 02:30:13.344339 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342312 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lzg2\" (UniqueName: \"kubernetes.io/projected/4b77648c-d60e-4ba3-806f-ea2f240ac764-kube-api-access-4lzg2\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.344339 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342428 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bddd2af-4d25-4c54-8f2d-800934a15ab2-system-cni-dir\") pod \"multus-additional-cni-plugins-2ddl4\" (UID: \"7bddd2af-4d25-4c54-8f2d-800934a15ab2\") " pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.344339 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-etc-tuned\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.344339 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342499 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0fea0ae-0424-4574-b264-04ea4bcdea35-dbus\") pod \"global-pull-secret-syncer-4fb84\" (UID: \"e0fea0ae-0424-4574-b264-04ea4bcdea35\") " pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:13.344339 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342529 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-lib-modules\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.344339 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342541 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bddd2af-4d25-4c54-8f2d-800934a15ab2-system-cni-dir\") pod \"multus-additional-cni-plugins-2ddl4\" (UID: \"7bddd2af-4d25-4c54-8f2d-800934a15ab2\") " pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.344339 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342553 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-var-lib-openvswitch\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.344339 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342577 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-host-cni-netd\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.344339 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342602 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6022d651-c68e-42a6-86c9-eee48a15f8a2-konnectivity-ca\") pod \"konnectivity-agent-9jpwt\" (UID: \"6022d651-c68e-42a6-86c9-eee48a15f8a2\") " pod="kube-system/konnectivity-agent-9jpwt" Sep 09 02:30:13.344339 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342608 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-var-lib-openvswitch\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.344339 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342640 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-host-run-k8s-cni-cncf-io\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.344339 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342670 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-host-run-netns\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.344339 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342705 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-multus-conf-dir\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.345032 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342724 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63637b1b-65f1-4df2-b182-6a673b4f9d6d-ovnkube-script-lib\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.345032 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342752 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-multus-conf-dir\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.345032 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342759 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-host-run-k8s-cni-cncf-io\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.345032 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342788 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0fea0ae-0424-4574-b264-04ea4bcdea35-dbus\") pod \"global-pull-secret-syncer-4fb84\" (UID: \"e0fea0ae-0424-4574-b264-04ea4bcdea35\") " pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:13.345032 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342792 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-host-run-netns\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.345032 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-etc-sysctl-d\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.345032 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342823 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4b77648c-d60e-4ba3-806f-ea2f240ac764-multus-daemon-config\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.345032 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342837 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-host-cni-netd\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.345032 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342845 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-var-lib-kubelet\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.345032 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342900 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-etc-sysctl-d\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.345032 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-run-openvswitch\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.345032 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342934 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63637b1b-65f1-4df2-b182-6a673b4f9d6d-ovnkube-config\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.345032 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342939 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-lib-modules\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.345032 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342966 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-var-lib-kubelet\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.345032 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.342968 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/326af4c7-6a8d-4510-92f6-36ba2efe0943-iptables-alerter-script\") pod \"iptables-alerter-wlxqx\" (UID: \"326af4c7-6a8d-4510-92f6-36ba2efe0943\") " pod="openshift-network-operator/iptables-alerter-wlxqx" Sep 09 02:30:13.345032 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.343057 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-system-cni-dir\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.345032 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.343089 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-run-openvswitch\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.345032 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.343123 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-host-var-lib-cni-bin\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.345672 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.343153 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6022d651-c68e-42a6-86c9-eee48a15f8a2-konnectivity-ca\") pod \"konnectivity-agent-9jpwt\" (UID: \"6022d651-c68e-42a6-86c9-eee48a15f8a2\") " pod="kube-system/konnectivity-agent-9jpwt" Sep 09 02:30:13.345672 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.343170 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-system-cni-dir\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.345672 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.343197 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-hostroot\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.345672 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.343223 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-etc-modprobe-d\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.345672 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.343229 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-host-var-lib-cni-bin\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.345672 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.343249 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-etc-systemd\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.345672 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.343283 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4b77648c-d60e-4ba3-806f-ea2f240ac764-hostroot\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.345672 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.343291 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret\") pod \"global-pull-secret-syncer-4fb84\" (UID: \"e0fea0ae-0424-4574-b264-04ea4bcdea35\") " pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:13.345672 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.343320 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b77648c-d60e-4ba3-806f-ea2f240ac764-cni-binary-copy\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.345672 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.343327 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-etc-modprobe-d\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.345672 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.343343 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-etc-systemd\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.345672 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.343361 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-tmp\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.345672 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.343387 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-systemd-units\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.345672 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:13.343426 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:13.345672 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.343469 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63637b1b-65f1-4df2-b182-6a673b4f9d6d-systemd-units\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.345672 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.343491 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63637b1b-65f1-4df2-b182-6a673b4f9d6d-ovnkube-config\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.345672 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:13.343505 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret podName:e0fea0ae-0424-4574-b264-04ea4bcdea35 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:13.843481536 +0000 UTC m=+1.973723637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret") pod "global-pull-secret-syncer-4fb84" (UID: "e0fea0ae-0424-4574-b264-04ea4bcdea35") : object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:13.346230 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.343579 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/326af4c7-6a8d-4510-92f6-36ba2efe0943-iptables-alerter-script\") pod \"iptables-alerter-wlxqx\" (UID: \"326af4c7-6a8d-4510-92f6-36ba2efe0943\") " pod="openshift-network-operator/iptables-alerter-wlxqx" Sep 09 02:30:13.346230 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.343801 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b77648c-d60e-4ba3-806f-ea2f240ac764-cni-binary-copy\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.346230 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.345030 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63637b1b-65f1-4df2-b182-6a673b4f9d6d-ovn-node-metrics-cert\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.346230 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.345161 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6022d651-c68e-42a6-86c9-eee48a15f8a2-agent-certs\") pod \"konnectivity-agent-9jpwt\" (UID: \"6022d651-c68e-42a6-86c9-eee48a15f8a2\") " pod="kube-system/konnectivity-agent-9jpwt" Sep 09 02:30:13.346230 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.345641 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-tmp\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.346230 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.345706 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-etc-tuned\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.356890 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.356867 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vphzl\" (UniqueName: \"kubernetes.io/projected/326af4c7-6a8d-4510-92f6-36ba2efe0943-kube-api-access-vphzl\") pod \"iptables-alerter-wlxqx\" (UID: \"326af4c7-6a8d-4510-92f6-36ba2efe0943\") " pod="openshift-network-operator/iptables-alerter-wlxqx" Sep 09 02:30:13.357318 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.357278 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2d5c\" (UniqueName: \"kubernetes.io/projected/7bddd2af-4d25-4c54-8f2d-800934a15ab2-kube-api-access-d2d5c\") pod \"multus-additional-cni-plugins-2ddl4\" (UID: \"7bddd2af-4d25-4c54-8f2d-800934a15ab2\") " pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.357627 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.357597 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9wg8\" (UniqueName: \"kubernetes.io/projected/63637b1b-65f1-4df2-b182-6a673b4f9d6d-kube-api-access-b9wg8\") pod \"ovnkube-node-t8g9x\" (UID: \"63637b1b-65f1-4df2-b182-6a673b4f9d6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.359167 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.359134 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpwqr\" (UniqueName: \"kubernetes.io/projected/2cfd0ce1-cffe-43a7-832b-3e845b5d391d-kube-api-access-gpwqr\") pod \"aws-ebs-csi-driver-node-s8gnd\" (UID: \"2cfd0ce1-cffe-43a7-832b-3e845b5d391d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.359821 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.359796 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbpsx\" (UniqueName: \"kubernetes.io/projected/40631b35-00b4-4a4b-ab2e-f769434da1f1-kube-api-access-hbpsx\") pod \"node-resolver-wr95f\" (UID: \"40631b35-00b4-4a4b-ab2e-f769434da1f1\") " pod="openshift-dns/node-resolver-wr95f" Sep 09 02:30:13.359942 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.359876 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9bv8\" (UniqueName: \"kubernetes.io/projected/2f959dac-774e-473c-b318-7cec8464e980-kube-api-access-z9bv8\") pod \"node-ca-qfvvm\" (UID: \"2f959dac-774e-473c-b318-7cec8464e980\") " pod="openshift-image-registry/node-ca-qfvvm" Sep 09 02:30:13.360096 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.360078 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lzg2\" (UniqueName: \"kubernetes.io/projected/4b77648c-d60e-4ba3-806f-ea2f240ac764-kube-api-access-4lzg2\") pod \"multus-9pqdb\" (UID: \"4b77648c-d60e-4ba3-806f-ea2f240ac764\") " pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.360218 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.360195 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6jqc\" (UniqueName: \"kubernetes.io/projected/2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a-kube-api-access-s6jqc\") pod \"tuned-6mzm8\" (UID: \"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a\") " pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.384250 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.384114 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qfvvm" Sep 09 02:30:13.389583 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.389565 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2ddl4" Sep 09 02:30:13.443926 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.443893 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs\") pod \"network-metrics-daemon-n7lh6\" (UID: \"59a52c60-f713-4a2f-8de1-5d7a2d43ac41\") " pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:13.444156 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:13.444133 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:13.444227 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:13.444215 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs podName:59a52c60-f713-4a2f-8de1-5d7a2d43ac41 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:13.944192806 +0000 UTC m=+2.074434883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs") pod "network-metrics-daemon-n7lh6" (UID: "59a52c60-f713-4a2f-8de1-5d7a2d43ac41") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:13.444386 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.444360 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjbwq\" (UniqueName: \"kubernetes.io/projected/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-kube-api-access-pjbwq\") pod \"network-metrics-daemon-n7lh6\" (UID: \"59a52c60-f713-4a2f-8de1-5d7a2d43ac41\") " pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:13.444515 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.444458 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlt7n\" (UniqueName: \"kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n\") pod \"network-check-target-zr2ql\" (UID: \"6e6ad2e4-55a9-4509-99b4-30958e61e3a6\") " pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:13.459105 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:13.459048 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 02:30:13.459105 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:13.459069 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 02:30:13.459105 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:13.459080 2577 projected.go:196] Error preparing data for projected volume kube-api-access-dlt7n for pod openshift-network-diagnostics/network-check-target-zr2ql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:13.459324 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:13.459146 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n podName:6e6ad2e4-55a9-4509-99b4-30958e61e3a6 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:13.959132024 +0000 UTC m=+2.089374100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dlt7n" (UniqueName: "kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n") pod "network-check-target-zr2ql" (UID: "6e6ad2e4-55a9-4509-99b4-30958e61e3a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:13.460713 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.460692 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjbwq\" (UniqueName: \"kubernetes.io/projected/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-kube-api-access-pjbwq\") pod \"network-metrics-daemon-n7lh6\" (UID: \"59a52c60-f713-4a2f-8de1-5d7a2d43ac41\") " pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:13.502558 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:13.502532 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a47394d5f75f79888655dda3114eff2.slice/crio-2778efedc38a10647c71d9f442e07c4d776af2811d034dceb44c7f5fff932808 WatchSource:0}: Error finding container 2778efedc38a10647c71d9f442e07c4d776af2811d034dceb44c7f5fff932808: Status 404 returned error can't find the container with id 2778efedc38a10647c71d9f442e07c4d776af2811d034dceb44c7f5fff932808 Sep 09 02:30:13.502885 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:13.502863 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f11ba69470c54c65a13a194197ca4c.slice/crio-27b46747e109f8e70c122b54cf3eda98409a0a1908fdde8cb1e6b2c437dd294d WatchSource:0}: Error finding container 27b46747e109f8e70c122b54cf3eda98409a0a1908fdde8cb1e6b2c437dd294d: Status 404 returned error can't find the container with id 27b46747e109f8e70c122b54cf3eda98409a0a1908fdde8cb1e6b2c437dd294d Sep 09 02:30:13.506438 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.506418 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 02:30:13.540106 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.540078 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wlxqx" Sep 09 02:30:13.547716 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:13.547694 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod326af4c7_6a8d_4510_92f6_36ba2efe0943.slice/crio-fc5e8b1702ce5fccc0e78114cb16e282e232fce542308991cefa8ea5e2ac15cb WatchSource:0}: Error finding container fc5e8b1702ce5fccc0e78114cb16e282e232fce542308991cefa8ea5e2ac15cb: Status 404 returned error can't find the container with id fc5e8b1702ce5fccc0e78114cb16e282e232fce542308991cefa8ea5e2ac15cb Sep 09 02:30:13.552978 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.552957 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" Sep 09 02:30:13.560615 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:13.560595 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cfd0ce1_cffe_43a7_832b_3e845b5d391d.slice/crio-99ab23a573c2a61aa7d9548abd909482fe5a0d27e7dc43f241b807c9bf1e5cae WatchSource:0}: Error finding container 99ab23a573c2a61aa7d9548abd909482fe5a0d27e7dc43f241b807c9bf1e5cae: Status 404 returned error can't find the container with id 99ab23a573c2a61aa7d9548abd909482fe5a0d27e7dc43f241b807c9bf1e5cae Sep 09 02:30:13.582068 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.582047 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" Sep 09 02:30:13.590301 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:13.590279 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b83b4ab_b231_41d3_a4e0_14dfa81f7e0a.slice/crio-81b88ac1678fc48679b3766479f41e7cacb2a8cca2a842bf29c88adc0684e35f WatchSource:0}: Error finding container 81b88ac1678fc48679b3766479f41e7cacb2a8cca2a842bf29c88adc0684e35f: Status 404 returned error can't find the container with id 81b88ac1678fc48679b3766479f41e7cacb2a8cca2a842bf29c88adc0684e35f Sep 09 02:30:13.602003 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.601984 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9pqdb" Sep 09 02:30:13.608966 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:13.608943 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b77648c_d60e_4ba3_806f_ea2f240ac764.slice/crio-7e54537295f839afed26ba17ef813b39add023d135555105e25b4158c813e49e WatchSource:0}: Error finding container 7e54537295f839afed26ba17ef813b39add023d135555105e25b4158c813e49e: Status 404 returned error can't find the container with id 7e54537295f839afed26ba17ef813b39add023d135555105e25b4158c813e49e Sep 09 02:30:13.619257 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.619230 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:13.626413 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:13.626393 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63637b1b_65f1_4df2_b182_6a673b4f9d6d.slice/crio-4133f92b1082b7cb97816463ac4610d594f9fd96ce0e44e123bb98bf3f9e1fbb WatchSource:0}: Error finding container 4133f92b1082b7cb97816463ac4610d594f9fd96ce0e44e123bb98bf3f9e1fbb: Status 404 returned error can't find the container with id 4133f92b1082b7cb97816463ac4610d594f9fd96ce0e44e123bb98bf3f9e1fbb Sep 09 02:30:13.640139 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.640111 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9jpwt" Sep 09 02:30:13.646824 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:13.646803 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6022d651_c68e_42a6_86c9_eee48a15f8a2.slice/crio-612a91e23014fc61599eada6559309b3c1378fa085b5763fb793d88ed2ff4386 WatchSource:0}: Error finding container 612a91e23014fc61599eada6559309b3c1378fa085b5763fb793d88ed2ff4386: Status 404 returned error can't find the container with id 612a91e23014fc61599eada6559309b3c1378fa085b5763fb793d88ed2ff4386 Sep 09 02:30:13.656970 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.656951 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wr95f" Sep 09 02:30:13.664406 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:13.664385 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40631b35_00b4_4a4b_ab2e_f769434da1f1.slice/crio-d1d3cc3ece5cdc529b67dfbc5dfe2c29e7125cfe95957d93e945cc45670d6176 WatchSource:0}: Error finding container d1d3cc3ece5cdc529b67dfbc5dfe2c29e7125cfe95957d93e945cc45670d6176: Status 404 returned error can't find the container with id d1d3cc3ece5cdc529b67dfbc5dfe2c29e7125cfe95957d93e945cc45670d6176 Sep 09 02:30:13.757552 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:13.757511 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f959dac_774e_473c_b318_7cec8464e980.slice/crio-7f6f89ef07a26d7c6aaf48348311b521047cc6c2c4c0ba47177af8414a019c03 WatchSource:0}: Error finding container 7f6f89ef07a26d7c6aaf48348311b521047cc6c2c4c0ba47177af8414a019c03: Status 404 returned error can't find the container with id 7f6f89ef07a26d7c6aaf48348311b521047cc6c2c4c0ba47177af8414a019c03 Sep 09 02:30:13.759263 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.759245 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 02:30:13.804061 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:13.804039 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bddd2af_4d25_4c54_8f2d_800934a15ab2.slice/crio-af4b119537f654cb617e464e45be3b0ceb5a5cc6d2070f87ebae07cf7a255c92 WatchSource:0}: Error finding container af4b119537f654cb617e464e45be3b0ceb5a5cc6d2070f87ebae07cf7a255c92: Status 404 returned error can't find the container with id af4b119537f654cb617e464e45be3b0ceb5a5cc6d2070f87ebae07cf7a255c92 Sep 09 02:30:13.845907 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.845879 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret\") pod \"global-pull-secret-syncer-4fb84\" (UID: \"e0fea0ae-0424-4574-b264-04ea4bcdea35\") " pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:13.846013 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:13.846000 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:13.846065 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:13.846049 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret podName:e0fea0ae-0424-4574-b264-04ea4bcdea35 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:14.846035563 +0000 UTC m=+2.976277639 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret") pod "global-pull-secret-syncer-4fb84" (UID: "e0fea0ae-0424-4574-b264-04ea4bcdea35") : object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:13.949355 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:13.949240 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs\") pod \"network-metrics-daemon-n7lh6\" (UID: \"59a52c60-f713-4a2f-8de1-5d7a2d43ac41\") " pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:13.949527 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:13.949396 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:13.949527 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:13.949474 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs podName:59a52c60-f713-4a2f-8de1-5d7a2d43ac41 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:14.949450561 +0000 UTC m=+3.079692638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs") pod "network-metrics-daemon-n7lh6" (UID: "59a52c60-f713-4a2f-8de1-5d7a2d43ac41") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:14.050425 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:14.050347 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlt7n\" (UniqueName: \"kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n\") pod \"network-check-target-zr2ql\" (UID: \"6e6ad2e4-55a9-4509-99b4-30958e61e3a6\") " pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:14.050589 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:14.050524 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 02:30:14.050589 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:14.050547 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 02:30:14.050589 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:14.050577 2577 projected.go:196] Error preparing data for projected volume kube-api-access-dlt7n for pod openshift-network-diagnostics/network-check-target-zr2ql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:14.050760 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:14.050636 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n podName:6e6ad2e4-55a9-4509-99b4-30958e61e3a6 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:15.050616885 +0000 UTC m=+3.180858978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dlt7n" (UniqueName: "kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n") pod "network-check-target-zr2ql" (UID: "6e6ad2e4-55a9-4509-99b4-30958e61e3a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:14.255341 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:14.255309 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 02:30:14.287403 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:14.286930 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 02:25:14 +0000 UTC" deadline="2028-03-11 12:15:19.777591502 +0000 UTC" Sep 09 02:30:14.287403 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:14.286986 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13185h45m5.490609288s" Sep 09 02:30:14.328304 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:14.327642 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:14.328304 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:14.327775 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2ql" podUID="6e6ad2e4-55a9-4509-99b4-30958e61e3a6" Sep 09 02:30:14.344478 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:14.344412 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" event={"ID":"2cfd0ce1-cffe-43a7-832b-3e845b5d391d","Type":"ContainerStarted","Data":"99ab23a573c2a61aa7d9548abd909482fe5a0d27e7dc43f241b807c9bf1e5cae"} Sep 09 02:30:14.360203 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:14.360166 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qfvvm" event={"ID":"2f959dac-774e-473c-b318-7cec8464e980","Type":"ContainerStarted","Data":"7f6f89ef07a26d7c6aaf48348311b521047cc6c2c4c0ba47177af8414a019c03"} Sep 09 02:30:14.374094 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:14.374015 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wr95f" event={"ID":"40631b35-00b4-4a4b-ab2e-f769434da1f1","Type":"ContainerStarted","Data":"d1d3cc3ece5cdc529b67dfbc5dfe2c29e7125cfe95957d93e945cc45670d6176"} Sep 09 02:30:14.384306 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:14.384277 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9pqdb" event={"ID":"4b77648c-d60e-4ba3-806f-ea2f240ac764","Type":"ContainerStarted","Data":"7e54537295f839afed26ba17ef813b39add023d135555105e25b4158c813e49e"} Sep 09 02:30:14.405299 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:14.405269 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" event={"ID":"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a","Type":"ContainerStarted","Data":"81b88ac1678fc48679b3766479f41e7cacb2a8cca2a842bf29c88adc0684e35f"} Sep 09 02:30:14.412575 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:14.412545 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wlxqx" event={"ID":"326af4c7-6a8d-4510-92f6-36ba2efe0943","Type":"ContainerStarted","Data":"fc5e8b1702ce5fccc0e78114cb16e282e232fce542308991cefa8ea5e2ac15cb"} Sep 09 02:30:14.428457 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:14.428389 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-37.ec2.internal" event={"ID":"7a47394d5f75f79888655dda3114eff2","Type":"ContainerStarted","Data":"2778efedc38a10647c71d9f442e07c4d776af2811d034dceb44c7f5fff932808"} Sep 09 02:30:14.447266 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:14.447233 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-37.ec2.internal" event={"ID":"60f11ba69470c54c65a13a194197ca4c","Type":"ContainerStarted","Data":"27b46747e109f8e70c122b54cf3eda98409a0a1908fdde8cb1e6b2c437dd294d"} Sep 09 02:30:14.467039 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:14.465638 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2ddl4" event={"ID":"7bddd2af-4d25-4c54-8f2d-800934a15ab2","Type":"ContainerStarted","Data":"af4b119537f654cb617e464e45be3b0ceb5a5cc6d2070f87ebae07cf7a255c92"} Sep 09 02:30:14.472245 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:14.472212 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9jpwt" event={"ID":"6022d651-c68e-42a6-86c9-eee48a15f8a2","Type":"ContainerStarted","Data":"612a91e23014fc61599eada6559309b3c1378fa085b5763fb793d88ed2ff4386"} Sep 09 02:30:14.497139 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:14.497109 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" event={"ID":"63637b1b-65f1-4df2-b182-6a673b4f9d6d","Type":"ContainerStarted","Data":"4133f92b1082b7cb97816463ac4610d594f9fd96ce0e44e123bb98bf3f9e1fbb"} Sep 09 02:30:14.857255 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:14.857211 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret\") pod \"global-pull-secret-syncer-4fb84\" (UID: \"e0fea0ae-0424-4574-b264-04ea4bcdea35\") " pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:14.857454 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:14.857352 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:14.857454 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:14.857420 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret podName:e0fea0ae-0424-4574-b264-04ea4bcdea35 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:16.857401263 +0000 UTC m=+4.987643341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret") pod "global-pull-secret-syncer-4fb84" (UID: "e0fea0ae-0424-4574-b264-04ea4bcdea35") : object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:14.959171 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:14.958538 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs\") pod \"network-metrics-daemon-n7lh6\" (UID: \"59a52c60-f713-4a2f-8de1-5d7a2d43ac41\") " pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:14.959171 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:14.958740 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:14.959171 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:14.958805 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs podName:59a52c60-f713-4a2f-8de1-5d7a2d43ac41 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:16.958785232 +0000 UTC m=+5.089027313 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs") pod "network-metrics-daemon-n7lh6" (UID: "59a52c60-f713-4a2f-8de1-5d7a2d43ac41") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:15.059824 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:15.059120 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlt7n\" (UniqueName: \"kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n\") pod \"network-check-target-zr2ql\" (UID: \"6e6ad2e4-55a9-4509-99b4-30958e61e3a6\") " pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:15.059824 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:15.059349 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 02:30:15.059824 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:15.059370 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 02:30:15.059824 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:15.059384 2577 projected.go:196] Error preparing data for projected volume kube-api-access-dlt7n for pod openshift-network-diagnostics/network-check-target-zr2ql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:15.059824 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:15.059447 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n podName:6e6ad2e4-55a9-4509-99b4-30958e61e3a6 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:17.059427876 +0000 UTC m=+5.189669957 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dlt7n" (UniqueName: "kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n") pod "network-check-target-zr2ql" (UID: "6e6ad2e4-55a9-4509-99b4-30958e61e3a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:15.288081 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:15.287486 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 02:25:14 +0000 UTC" deadline="2028-03-10 21:52:50.800699839 +0000 UTC" Sep 09 02:30:15.288081 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:15.287544 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13171h22m35.513159634s" Sep 09 02:30:15.304033 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:15.304003 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:15.304194 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:15.304136 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fb84" podUID="e0fea0ae-0424-4574-b264-04ea4bcdea35" Sep 09 02:30:15.304590 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:15.304571 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:15.304707 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:15.304685 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7lh6" podUID="59a52c60-f713-4a2f-8de1-5d7a2d43ac41" Sep 09 02:30:16.308964 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:16.306354 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:16.308964 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:16.306495 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2ql" podUID="6e6ad2e4-55a9-4509-99b4-30958e61e3a6" Sep 09 02:30:16.878120 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:16.878084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret\") pod \"global-pull-secret-syncer-4fb84\" (UID: \"e0fea0ae-0424-4574-b264-04ea4bcdea35\") " pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:16.878326 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:16.878279 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:16.878405 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:16.878343 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret podName:e0fea0ae-0424-4574-b264-04ea4bcdea35 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:20.878323278 +0000 UTC m=+9.008565361 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret") pod "global-pull-secret-syncer-4fb84" (UID: "e0fea0ae-0424-4574-b264-04ea4bcdea35") : object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:16.979577 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:16.978887 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs\") pod \"network-metrics-daemon-n7lh6\" (UID: \"59a52c60-f713-4a2f-8de1-5d7a2d43ac41\") " pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:16.979577 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:16.979082 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:16.979577 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:16.979169 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs podName:59a52c60-f713-4a2f-8de1-5d7a2d43ac41 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:20.979147924 +0000 UTC m=+9.109390004 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs") pod "network-metrics-daemon-n7lh6" (UID: "59a52c60-f713-4a2f-8de1-5d7a2d43ac41") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:17.079655 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:17.079619 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlt7n\" (UniqueName: \"kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n\") pod \"network-check-target-zr2ql\" (UID: \"6e6ad2e4-55a9-4509-99b4-30958e61e3a6\") " pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:17.079882 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:17.079826 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 02:30:17.079882 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:17.079847 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 02:30:17.079882 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:17.079873 2577 projected.go:196] Error preparing data for projected volume kube-api-access-dlt7n for pod openshift-network-diagnostics/network-check-target-zr2ql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:17.080121 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:17.079943 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n podName:6e6ad2e4-55a9-4509-99b4-30958e61e3a6 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:21.079914629 +0000 UTC m=+9.210156734 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dlt7n" (UniqueName: "kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n") pod "network-check-target-zr2ql" (UID: "6e6ad2e4-55a9-4509-99b4-30958e61e3a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:17.304411 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:17.304319 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:17.304624 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:17.304461 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fb84" podUID="e0fea0ae-0424-4574-b264-04ea4bcdea35" Sep 09 02:30:17.304923 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:17.304896 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:17.305099 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:17.305077 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7lh6" podUID="59a52c60-f713-4a2f-8de1-5d7a2d43ac41" Sep 09 02:30:18.308422 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:18.307898 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:18.308422 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:18.308045 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2ql" podUID="6e6ad2e4-55a9-4509-99b4-30958e61e3a6" Sep 09 02:30:19.304457 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:19.304377 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:19.304652 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:19.304495 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fb84" podUID="e0fea0ae-0424-4574-b264-04ea4bcdea35" Sep 09 02:30:19.304870 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:19.304844 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:19.305005 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:19.304974 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7lh6" podUID="59a52c60-f713-4a2f-8de1-5d7a2d43ac41" Sep 09 02:30:20.303634 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:20.303598 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:20.304100 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:20.303738 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2ql" podUID="6e6ad2e4-55a9-4509-99b4-30958e61e3a6" Sep 09 02:30:20.915288 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:20.915137 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret\") pod \"global-pull-secret-syncer-4fb84\" (UID: \"e0fea0ae-0424-4574-b264-04ea4bcdea35\") " pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:20.915496 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:20.915411 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:20.915567 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:20.915505 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret podName:e0fea0ae-0424-4574-b264-04ea4bcdea35 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:28.915481143 +0000 UTC m=+17.045723241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret") pod "global-pull-secret-syncer-4fb84" (UID: "e0fea0ae-0424-4574-b264-04ea4bcdea35") : object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:21.016283 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:21.016244 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs\") pod \"network-metrics-daemon-n7lh6\" (UID: \"59a52c60-f713-4a2f-8de1-5d7a2d43ac41\") " pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:21.016476 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:21.016444 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:21.016547 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:21.016539 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs podName:59a52c60-f713-4a2f-8de1-5d7a2d43ac41 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:29.016515507 +0000 UTC m=+17.146757585 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs") pod "network-metrics-daemon-n7lh6" (UID: "59a52c60-f713-4a2f-8de1-5d7a2d43ac41") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:21.117195 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:21.117150 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlt7n\" (UniqueName: \"kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n\") pod \"network-check-target-zr2ql\" (UID: \"6e6ad2e4-55a9-4509-99b4-30958e61e3a6\") " pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:21.117395 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:21.117372 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 02:30:21.117470 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:21.117404 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 02:30:21.117470 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:21.117422 2577 projected.go:196] Error preparing data for projected volume kube-api-access-dlt7n for pod openshift-network-diagnostics/network-check-target-zr2ql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:21.117592 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:21.117493 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n podName:6e6ad2e4-55a9-4509-99b4-30958e61e3a6 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:29.117472635 +0000 UTC m=+17.247714726 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dlt7n" (UniqueName: "kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n") pod "network-check-target-zr2ql" (UID: "6e6ad2e4-55a9-4509-99b4-30958e61e3a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:21.303772 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:21.303680 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:21.304390 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:21.303859 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7lh6" podUID="59a52c60-f713-4a2f-8de1-5d7a2d43ac41" Sep 09 02:30:21.304390 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:21.304260 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:21.304390 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:21.304357 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fb84" podUID="e0fea0ae-0424-4574-b264-04ea4bcdea35" Sep 09 02:30:22.307656 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:22.307625 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:22.308173 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:22.307768 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2ql" podUID="6e6ad2e4-55a9-4509-99b4-30958e61e3a6" Sep 09 02:30:23.304188 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:23.304150 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:23.304394 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:23.304190 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:23.304394 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:23.304287 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fb84" podUID="e0fea0ae-0424-4574-b264-04ea4bcdea35" Sep 09 02:30:23.304582 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:23.304540 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7lh6" podUID="59a52c60-f713-4a2f-8de1-5d7a2d43ac41" Sep 09 02:30:24.303446 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:24.303413 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:24.303979 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:24.303547 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2ql" podUID="6e6ad2e4-55a9-4509-99b4-30958e61e3a6" Sep 09 02:30:25.303465 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:25.303429 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:25.303900 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:25.303429 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:25.303900 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:25.303567 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7lh6" podUID="59a52c60-f713-4a2f-8de1-5d7a2d43ac41" Sep 09 02:30:25.303900 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:25.303636 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fb84" podUID="e0fea0ae-0424-4574-b264-04ea4bcdea35" Sep 09 02:30:26.303884 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:26.303845 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:26.304325 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:26.303985 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2ql" podUID="6e6ad2e4-55a9-4509-99b4-30958e61e3a6" Sep 09 02:30:27.304376 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:27.304333 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:27.304862 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:27.304387 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:27.304862 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:27.304487 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7lh6" podUID="59a52c60-f713-4a2f-8de1-5d7a2d43ac41" Sep 09 02:30:27.304862 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:27.304602 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fb84" podUID="e0fea0ae-0424-4574-b264-04ea4bcdea35" Sep 09 02:30:28.304397 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:28.304350 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:28.304887 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:28.304476 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2ql" podUID="6e6ad2e4-55a9-4509-99b4-30958e61e3a6" Sep 09 02:30:28.974663 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:28.974613 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret\") pod \"global-pull-secret-syncer-4fb84\" (UID: \"e0fea0ae-0424-4574-b264-04ea4bcdea35\") " pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:28.974886 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:28.974771 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:28.974886 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:28.974853 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret podName:e0fea0ae-0424-4574-b264-04ea4bcdea35 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:44.974831585 +0000 UTC m=+33.105073673 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret") pod "global-pull-secret-syncer-4fb84" (UID: "e0fea0ae-0424-4574-b264-04ea4bcdea35") : object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:29.075701 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:29.075664 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs\") pod \"network-metrics-daemon-n7lh6\" (UID: \"59a52c60-f713-4a2f-8de1-5d7a2d43ac41\") " pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:29.075919 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:29.075836 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:29.075919 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:29.075915 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs podName:59a52c60-f713-4a2f-8de1-5d7a2d43ac41 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:45.075894585 +0000 UTC m=+33.206136681 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs") pod "network-metrics-daemon-n7lh6" (UID: "59a52c60-f713-4a2f-8de1-5d7a2d43ac41") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:29.176533 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:29.176497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlt7n\" (UniqueName: \"kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n\") pod \"network-check-target-zr2ql\" (UID: \"6e6ad2e4-55a9-4509-99b4-30958e61e3a6\") " pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:29.176700 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:29.176666 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 02:30:29.176700 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:29.176684 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 02:30:29.176700 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:29.176693 2577 projected.go:196] Error preparing data for projected volume kube-api-access-dlt7n for pod openshift-network-diagnostics/network-check-target-zr2ql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:29.176865 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:29.176743 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n podName:6e6ad2e4-55a9-4509-99b4-30958e61e3a6 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:45.176726388 +0000 UTC m=+33.306968463 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dlt7n" (UniqueName: "kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n") pod "network-check-target-zr2ql" (UID: "6e6ad2e4-55a9-4509-99b4-30958e61e3a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:29.303626 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:29.303532 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:29.303626 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:29.303557 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:29.303931 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:29.303657 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fb84" podUID="e0fea0ae-0424-4574-b264-04ea4bcdea35" Sep 09 02:30:29.303931 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:29.303807 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7lh6" podUID="59a52c60-f713-4a2f-8de1-5d7a2d43ac41" Sep 09 02:30:30.304347 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:30.304316 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:30.304941 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:30.304434 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2ql" podUID="6e6ad2e4-55a9-4509-99b4-30958e61e3a6" Sep 09 02:30:31.303731 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:31.303690 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:31.303731 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:31.303718 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:31.304022 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:31.303837 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fb84" podUID="e0fea0ae-0424-4574-b264-04ea4bcdea35" Sep 09 02:30:31.304022 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:31.303930 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7lh6" podUID="59a52c60-f713-4a2f-8de1-5d7a2d43ac41" Sep 09 02:30:32.304112 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:32.304079 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:32.304512 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:32.304168 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2ql" podUID="6e6ad2e4-55a9-4509-99b4-30958e61e3a6" Sep 09 02:30:33.304475 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:33.304250 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:33.304865 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:33.304298 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:33.304865 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:33.304591 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fb84" podUID="e0fea0ae-0424-4574-b264-04ea4bcdea35" Sep 09 02:30:33.304865 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:33.304680 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7lh6" podUID="59a52c60-f713-4a2f-8de1-5d7a2d43ac41" Sep 09 02:30:33.553635 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:33.553601 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9pqdb" event={"ID":"4b77648c-d60e-4ba3-806f-ea2f240ac764","Type":"ContainerStarted","Data":"6db554fe92855f84b7c14611df8972783bdbebc7bf29607174b6387d067d8edf"} Sep 09 02:30:33.555052 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:33.555030 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" event={"ID":"2b83b4ab-b231-41d3-a4e0-14dfa81f7e0a","Type":"ContainerStarted","Data":"16938248699d36322e0b951c3304c7a68cd3f198f0584acb450469c0c26f2d36"} Sep 09 02:30:33.556424 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:33.556404 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-37.ec2.internal" event={"ID":"60f11ba69470c54c65a13a194197ca4c","Type":"ContainerStarted","Data":"241e596874cb9570f087dbd0246f92ab7374b3031155f84f70fd1c2c6c8cecc7"} Sep 09 02:30:33.587726 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:33.587687 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9pqdb" podStartSLOduration=1.026989978 podStartE2EDuration="20.587676814s" totalImagesPullingTime="19.560686836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:33.584904669 +0000 UTC m=+21.715146765" watchObservedRunningTime="2026-09-09 02:30:33.587676814 +0000 UTC m=+21.717918909" Sep 09 02:30:33.619235 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:33.619198 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6mzm8" podStartSLOduration=1.06782985 podStartE2EDuration="20.619184877s" totalImagesPullingTime="19.551355027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:33.619173758 +0000 UTC m=+21.749415853" watchObservedRunningTime="2026-09-09 02:30:33.619184877 +0000 UTC m=+21.749426972" Sep 09 02:30:33.649150 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:33.649078 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-37.ec2.internal" podStartSLOduration=19.649064129 podStartE2EDuration="19.649064129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:33.648813113 +0000 UTC m=+21.779055210" watchObservedRunningTime="2026-09-09 02:30:33.649064129 +0000 UTC m=+21.779306225" Sep 09 02:30:34.303563 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:34.303337 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:34.303680 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:34.303658 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2ql" podUID="6e6ad2e4-55a9-4509-99b4-30958e61e3a6" Sep 09 02:30:34.560276 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:34.560244 2577 generic.go:356] "Generic (PLEG): container finished" podID="7a47394d5f75f79888655dda3114eff2" containerID="f79aba1abe0f02c8593947a5679f6fce7de3d46004a1006af5c2e2d59cb739e7" exitCode=0 Sep 09 02:30:34.561055 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:34.560318 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-37.ec2.internal" event={"ID":"7a47394d5f75f79888655dda3114eff2","Type":"ContainerDied","Data":"f79aba1abe0f02c8593947a5679f6fce7de3d46004a1006af5c2e2d59cb739e7"} Sep 09 02:30:34.561968 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:34.561946 2577 generic.go:356] "Generic (PLEG): container finished" podID="7bddd2af-4d25-4c54-8f2d-800934a15ab2" containerID="ec83fbe460f94b39781b1b671994acb4cc2202e86d3f7f240d7c2a182eb1e3ba" exitCode=0 Sep 09 02:30:34.562090 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:34.562025 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2ddl4" event={"ID":"7bddd2af-4d25-4c54-8f2d-800934a15ab2","Type":"ContainerDied","Data":"ec83fbe460f94b39781b1b671994acb4cc2202e86d3f7f240d7c2a182eb1e3ba"} Sep 09 02:30:34.563559 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:34.563535 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9jpwt" event={"ID":"6022d651-c68e-42a6-86c9-eee48a15f8a2","Type":"ContainerStarted","Data":"81a00ff9d4589bc228aae36e42ed2d690070aa63cbe0da550c9cda2c0fd82586"} Sep 09 02:30:34.569967 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:34.569945 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" event={"ID":"63637b1b-65f1-4df2-b182-6a673b4f9d6d","Type":"ContainerStarted","Data":"ba976302250f611bbd8ae79f009c46ce2c0ddca2241827e7700fc861805fde73"} Sep 09 02:30:34.570103 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:34.570030 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" event={"ID":"63637b1b-65f1-4df2-b182-6a673b4f9d6d","Type":"ContainerStarted","Data":"ed2f450e6cb8b1ee0ab3521a05ee1485bee38251403dc211861a9383c7ec2d98"} Sep 09 02:30:34.570103 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:34.570080 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" event={"ID":"63637b1b-65f1-4df2-b182-6a673b4f9d6d","Type":"ContainerStarted","Data":"720168c722929c7e0bb5cf07f41d8a471764cb019f9bf9ff6e63d970542af08e"} Sep 09 02:30:34.570202 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:34.570130 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" event={"ID":"63637b1b-65f1-4df2-b182-6a673b4f9d6d","Type":"ContainerStarted","Data":"d473912795489892e946751388216925fdff6a96dfd738ce7dcdf9c2115eaf45"} Sep 09 02:30:34.570202 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:34.570180 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" event={"ID":"63637b1b-65f1-4df2-b182-6a673b4f9d6d","Type":"ContainerStarted","Data":"5108b7321fbe3e6a8096fcc79287413ca939ae536db0c2d73f1648414d41496b"} Sep 09 02:30:34.570289 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:34.570212 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" event={"ID":"63637b1b-65f1-4df2-b182-6a673b4f9d6d","Type":"ContainerStarted","Data":"316cf3440eefcea07ff3242c249850589aa59513ab960a1440438f92491dad0c"} Sep 09 02:30:34.571587 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:34.571567 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" event={"ID":"2cfd0ce1-cffe-43a7-832b-3e845b5d391d","Type":"ContainerStarted","Data":"ab785231e23bb6e96066db8b38a5c9a6bc255e13b25477ca283d06839ac48c64"} Sep 09 02:30:34.573074 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:34.573051 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qfvvm" event={"ID":"2f959dac-774e-473c-b318-7cec8464e980","Type":"ContainerStarted","Data":"772b1157bac6293871177c4df83ba9fbbb4aa1a339b0a0cd1a6eea753582f44b"} Sep 09 02:30:34.574632 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:34.574602 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wr95f" event={"ID":"40631b35-00b4-4a4b-ab2e-f769434da1f1","Type":"ContainerStarted","Data":"16fc7dda20a75f3b7755258b131219e55b9d716f86e3a0bbe6d07baadddcd76f"} Sep 09 02:30:34.606540 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:34.606498 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qfvvm" podStartSLOduration=2.222424417 podStartE2EDuration="21.606484139s" totalImagesPullingTime="19.384059722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:34.605943428 +0000 UTC m=+22.736185525" watchObservedRunningTime="2026-09-09 02:30:34.606484139 +0000 UTC m=+22.736726255" Sep 09 02:30:34.654950 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:34.654851 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9jpwt" podStartSLOduration=2.161444053 podStartE2EDuration="21.654808057s" totalImagesPullingTime="19.493364004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:34.634804904 +0000 UTC m=+22.765047001" watchObservedRunningTime="2026-09-09 02:30:34.654808057 +0000 UTC m=+22.785050158" Sep 09 02:30:34.655100 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:34.654961 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wr95f" podStartSLOduration=2.211938627 podStartE2EDuration="21.654955796s" totalImagesPullingTime="19.443017169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:34.654031182 +0000 UTC m=+22.784273280" watchObservedRunningTime="2026-09-09 02:30:34.654955796 +0000 UTC m=+22.785197893" Sep 09 02:30:35.303912 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:35.303691 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:35.304049 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:35.303730 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:35.304101 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:35.304052 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7lh6" podUID="59a52c60-f713-4a2f-8de1-5d7a2d43ac41" Sep 09 02:30:35.304189 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:35.304171 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fb84" podUID="e0fea0ae-0424-4574-b264-04ea4bcdea35" Sep 09 02:30:35.335297 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:35.335212 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 09 02:30:35.438210 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:35.438174 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9jpwt" Sep 09 02:30:35.438987 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:35.438963 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9jpwt" Sep 09 02:30:35.448605 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:35.448513 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T02:30:35.335228014Z","UUID":"0170bb8b-ebea-4662-9d09-40db157b0c35","Handler":null,"Name":"","Endpoint":""} Sep 09 02:30:35.450285 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:35.450266 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 09 02:30:35.450383 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:35.450308 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 09 02:30:35.579155 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:35.579047 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" event={"ID":"2cfd0ce1-cffe-43a7-832b-3e845b5d391d","Type":"ContainerStarted","Data":"05e18a6f102a98e33cc465c1d203fa09e2cb57132a55297ffc710064edb19614"} Sep 09 02:30:35.580776 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:35.580694 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wlxqx" event={"ID":"326af4c7-6a8d-4510-92f6-36ba2efe0943","Type":"ContainerStarted","Data":"b1cb102f12b64f0e1bdceabd3ed0518e04187e721b65569a2f15748fe415af27"} Sep 09 02:30:36.303973 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:36.303941 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:36.304281 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:36.304066 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2ql" podUID="6e6ad2e4-55a9-4509-99b4-30958e61e3a6" Sep 09 02:30:36.597371 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:36.597278 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-37.ec2.internal" event={"ID":"7a47394d5f75f79888655dda3114eff2","Type":"ContainerStarted","Data":"8d25f5fff2e71850c5ced061a7a5eca271b2ca2748d912c389ad103ac3e48e38"} Sep 09 02:30:36.597806 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:36.597481 2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 09 02:30:36.615576 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:36.615535 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wlxqx" podStartSLOduration=4.018648173 podStartE2EDuration="23.615518168s" totalImagesPullingTime="19.596869995s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:35.620033749 +0000 UTC m=+23.750275846" watchObservedRunningTime="2026-09-09 02:30:36.615518168 +0000 UTC m=+24.745760263" Sep 09 02:30:37.303463 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:37.303433 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:37.303628 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:37.303434 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:37.303628 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:37.303534 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fb84" podUID="e0fea0ae-0424-4574-b264-04ea4bcdea35" Sep 09 02:30:37.303628 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:37.303621 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7lh6" podUID="59a52c60-f713-4a2f-8de1-5d7a2d43ac41" Sep 09 02:30:37.486328 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:37.486291 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9jpwt" Sep 09 02:30:37.486892 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:37.486868 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9jpwt" Sep 09 02:30:37.505915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:37.505866 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-37.ec2.internal" podStartSLOduration=23.505852219 podStartE2EDuration="23.505852219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:36.615273703 +0000 UTC m=+24.745515802" watchObservedRunningTime="2026-09-09 02:30:37.505852219 +0000 UTC m=+25.636094318" Sep 09 02:30:37.602002 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:37.601926 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" event={"ID":"2cfd0ce1-cffe-43a7-832b-3e845b5d391d","Type":"ContainerStarted","Data":"69c0eb91974b7d646bbcb0934d77d0722e3c3c3bbcf309e9c94dd89463b8e5dc"} Sep 09 02:30:37.605005 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:37.604977 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" event={"ID":"63637b1b-65f1-4df2-b182-6a673b4f9d6d","Type":"ContainerStarted","Data":"32401b3bb72073341cda30cce66d1e15882b88c31cf4a60b7933600453a30e18"} Sep 09 02:30:37.626033 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:37.625992 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8gnd" podStartSLOduration=2.667685884 podStartE2EDuration="24.625977468s" totalImagesPullingTime="21.958291584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:37.625781526 +0000 UTC m=+25.756023622" watchObservedRunningTime="2026-09-09 02:30:37.625977468 +0000 UTC m=+25.756219594" Sep 09 02:30:38.304279 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:38.304244 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:38.304490 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:38.304377 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2ql" podUID="6e6ad2e4-55a9-4509-99b4-30958e61e3a6" Sep 09 02:30:39.304391 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:39.304348 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:39.304391 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:39.304395 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:39.304877 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:39.304489 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7lh6" podUID="59a52c60-f713-4a2f-8de1-5d7a2d43ac41" Sep 09 02:30:39.304877 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:39.304648 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fb84" podUID="e0fea0ae-0424-4574-b264-04ea4bcdea35" Sep 09 02:30:40.304225 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:40.304023 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:40.304364 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:40.304252 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2ql" podUID="6e6ad2e4-55a9-4509-99b4-30958e61e3a6" Sep 09 02:30:40.613746 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:40.613676 2577 generic.go:356] "Generic (PLEG): container finished" podID="7bddd2af-4d25-4c54-8f2d-800934a15ab2" containerID="5abc28dbfbbae4a78c46cca7e686295d4d51248d788bb9659f6f3290413abf99" exitCode=0 Sep 09 02:30:40.614278 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:40.613768 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2ddl4" event={"ID":"7bddd2af-4d25-4c54-8f2d-800934a15ab2","Type":"ContainerDied","Data":"5abc28dbfbbae4a78c46cca7e686295d4d51248d788bb9659f6f3290413abf99"} Sep 09 02:30:40.617579 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:40.617557 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" event={"ID":"63637b1b-65f1-4df2-b182-6a673b4f9d6d","Type":"ContainerStarted","Data":"157ed6d9df41f44dd0c6bed37fd1ab719de67b8172a6cd291d591d3d88b443d7"} Sep 09 02:30:40.617903 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:40.617889 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:40.617977 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:40.617962 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:40.632992 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:40.632964 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:40.673240 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:40.673191 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" podStartSLOduration=7.525174091 podStartE2EDuration="27.673178543s" totalImagesPullingTime="20.148004452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:40.673018903 +0000 UTC m=+28.803261001" watchObservedRunningTime="2026-09-09 02:30:40.673178543 +0000 UTC m=+28.803420639" Sep 09 02:30:41.304304 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:41.304245 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:41.304448 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:41.304245 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:41.304448 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:41.304343 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fb84" podUID="e0fea0ae-0424-4574-b264-04ea4bcdea35" Sep 09 02:30:41.304448 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:41.304415 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7lh6" podUID="59a52c60-f713-4a2f-8de1-5d7a2d43ac41" Sep 09 02:30:41.619977 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:41.619913 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:41.637532 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:41.637207 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:30:41.884874 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:41.884636 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4fb84"] Sep 09 02:30:41.884973 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:41.884944 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:41.885045 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:41.885028 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fb84" podUID="e0fea0ae-0424-4574-b264-04ea4bcdea35" Sep 09 02:30:41.888148 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:41.888102 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zr2ql"] Sep 09 02:30:41.888286 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:41.888198 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:41.888286 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:41.888279 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2ql" podUID="6e6ad2e4-55a9-4509-99b4-30958e61e3a6" Sep 09 02:30:41.888884 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:41.888865 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n7lh6"] Sep 09 02:30:41.888971 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:41.888951 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:41.889058 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:41.889020 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7lh6" podUID="59a52c60-f713-4a2f-8de1-5d7a2d43ac41" Sep 09 02:30:42.623517 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:42.623488 2577 generic.go:356] "Generic (PLEG): container finished" podID="7bddd2af-4d25-4c54-8f2d-800934a15ab2" containerID="753e871a5d841416b4e0e919dd3d1ff6e93fb1e96aff6f001c3437ba41855689" exitCode=0 Sep 09 02:30:42.624153 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:42.623578 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2ddl4" event={"ID":"7bddd2af-4d25-4c54-8f2d-800934a15ab2","Type":"ContainerDied","Data":"753e871a5d841416b4e0e919dd3d1ff6e93fb1e96aff6f001c3437ba41855689"} Sep 09 02:30:43.245700 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:43.245578 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:30:43.303961 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:43.303926 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:43.304152 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:43.304052 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:43.304152 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:43.304058 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2ql" podUID="6e6ad2e4-55a9-4509-99b4-30958e61e3a6" Sep 09 02:30:43.304152 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:43.304092 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:43.304296 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:43.304207 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7lh6" podUID="59a52c60-f713-4a2f-8de1-5d7a2d43ac41" Sep 09 02:30:43.304359 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:43.304329 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fb84" podUID="e0fea0ae-0424-4574-b264-04ea4bcdea35" Sep 09 02:30:43.392423 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:43.392394 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:30:44.633123 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:44.633094 2577 generic.go:356] "Generic (PLEG): container finished" podID="7bddd2af-4d25-4c54-8f2d-800934a15ab2" containerID="2c7e6b065a0c254493bebf904ec0bd9674cdd4ffa2bfa7abdc9b6a6c784d3f1d" exitCode=0 Sep 09 02:30:44.633524 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:44.633153 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2ddl4" event={"ID":"7bddd2af-4d25-4c54-8f2d-800934a15ab2","Type":"ContainerDied","Data":"2c7e6b065a0c254493bebf904ec0bd9674cdd4ffa2bfa7abdc9b6a6c784d3f1d"} Sep 09 02:30:44.988265 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:44.988186 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret\") pod \"global-pull-secret-syncer-4fb84\" (UID: \"e0fea0ae-0424-4574-b264-04ea4bcdea35\") " pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:44.988435 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:44.988299 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:44.988435 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:44.988365 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret podName:e0fea0ae-0424-4574-b264-04ea4bcdea35 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:16.988350103 +0000 UTC m=+65.118592177 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret") pod "global-pull-secret-syncer-4fb84" (UID: "e0fea0ae-0424-4574-b264-04ea4bcdea35") : object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:45.089028 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:45.088998 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs\") pod \"network-metrics-daemon-n7lh6\" (UID: \"59a52c60-f713-4a2f-8de1-5d7a2d43ac41\") " pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:45.089189 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:45.089108 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:45.089189 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:45.089157 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs podName:59a52c60-f713-4a2f-8de1-5d7a2d43ac41 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:17.089145632 +0000 UTC m=+65.219387707 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs") pod "network-metrics-daemon-n7lh6" (UID: "59a52c60-f713-4a2f-8de1-5d7a2d43ac41") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:45.189677 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:45.189642 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlt7n\" (UniqueName: \"kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n\") pod \"network-check-target-zr2ql\" (UID: \"6e6ad2e4-55a9-4509-99b4-30958e61e3a6\") " pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:45.189876 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:45.189800 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 02:30:45.189876 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:45.189837 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 02:30:45.189876 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:45.189850 2577 projected.go:196] Error preparing data for projected volume kube-api-access-dlt7n for pod openshift-network-diagnostics/network-check-target-zr2ql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:45.190055 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:45.189912 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n podName:6e6ad2e4-55a9-4509-99b4-30958e61e3a6 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:17.189894196 +0000 UTC m=+65.320136272 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dlt7n" (UniqueName: "kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n") pod "network-check-target-zr2ql" (UID: "6e6ad2e4-55a9-4509-99b4-30958e61e3a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:45.303675 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:45.303584 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:45.303675 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:45.303629 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:45.303675 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:45.303629 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:45.303965 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:45.303712 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2ql" podUID="6e6ad2e4-55a9-4509-99b4-30958e61e3a6" Sep 09 02:30:45.303965 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:45.303847 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7lh6" podUID="59a52c60-f713-4a2f-8de1-5d7a2d43ac41" Sep 09 02:30:45.303965 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:45.303923 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fb84" podUID="e0fea0ae-0424-4574-b264-04ea4bcdea35" Sep 09 02:30:46.233401 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.233355 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:30:46.639032 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.638952 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:30:46.660960 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.660934 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-37.ec2.internal" event="NodeReady" Sep 09 02:30:46.661102 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.661049 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 09 02:30:46.725953 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.725903 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2zpcz"] Sep 09 02:30:46.740019 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.739989 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-z9xdr"] Sep 09 02:30:46.740194 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.740154 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2zpcz" Sep 09 02:30:46.742869 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.742832 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:30:46.743716 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.743698 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 09 02:30:46.744084 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.744066 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 09 02:30:46.744187 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.744067 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-99mnl\"" Sep 09 02:30:46.758536 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.758512 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z9xdr"] Sep 09 02:30:46.758653 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.758554 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2zpcz"] Sep 09 02:30:46.758653 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.758601 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z9xdr" Sep 09 02:30:46.761524 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.761502 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-r6wq8\"" Sep 09 02:30:46.762120 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.762098 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 09 02:30:46.762221 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.762102 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 09 02:30:46.762221 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.762137 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 09 02:30:46.902964 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.902883 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd4dj\" (UniqueName: \"kubernetes.io/projected/7aa94a6d-db29-4087-9416-92df99df14da-kube-api-access-gd4dj\") pod \"ingress-canary-z9xdr\" (UID: \"7aa94a6d-db29-4087-9416-92df99df14da\") " pod="openshift-ingress-canary/ingress-canary-z9xdr" Sep 09 02:30:46.903123 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.902976 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/35546a3a-6fe9-4cff-baae-84e4a07c527a-config-volume\") pod \"dns-default-2zpcz\" (UID: \"35546a3a-6fe9-4cff-baae-84e4a07c527a\") " pod="openshift-dns/dns-default-2zpcz" Sep 09 02:30:46.903123 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.903006 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7aa94a6d-db29-4087-9416-92df99df14da-cert\") pod \"ingress-canary-z9xdr\" (UID: \"7aa94a6d-db29-4087-9416-92df99df14da\") " pod="openshift-ingress-canary/ingress-canary-z9xdr" Sep 09 02:30:46.903123 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.903029 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35546a3a-6fe9-4cff-baae-84e4a07c527a-tmp-dir\") pod \"dns-default-2zpcz\" (UID: \"35546a3a-6fe9-4cff-baae-84e4a07c527a\") " pod="openshift-dns/dns-default-2zpcz" Sep 09 02:30:46.903280 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.903157 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35546a3a-6fe9-4cff-baae-84e4a07c527a-metrics-tls\") pod \"dns-default-2zpcz\" (UID: \"35546a3a-6fe9-4cff-baae-84e4a07c527a\") " pod="openshift-dns/dns-default-2zpcz" Sep 09 02:30:46.903280 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:46.903185 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8ksv\" (UniqueName: \"kubernetes.io/projected/35546a3a-6fe9-4cff-baae-84e4a07c527a-kube-api-access-q8ksv\") pod \"dns-default-2zpcz\" (UID: \"35546a3a-6fe9-4cff-baae-84e4a07c527a\") " pod="openshift-dns/dns-default-2zpcz" Sep 09 02:30:47.004263 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.004219 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/35546a3a-6fe9-4cff-baae-84e4a07c527a-config-volume\") pod \"dns-default-2zpcz\" (UID: \"35546a3a-6fe9-4cff-baae-84e4a07c527a\") " pod="openshift-dns/dns-default-2zpcz" Sep 09 02:30:47.004263 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.004270 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7aa94a6d-db29-4087-9416-92df99df14da-cert\") pod \"ingress-canary-z9xdr\" (UID: \"7aa94a6d-db29-4087-9416-92df99df14da\") " pod="openshift-ingress-canary/ingress-canary-z9xdr" Sep 09 02:30:47.004519 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.004292 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35546a3a-6fe9-4cff-baae-84e4a07c527a-tmp-dir\") pod \"dns-default-2zpcz\" (UID: \"35546a3a-6fe9-4cff-baae-84e4a07c527a\") " pod="openshift-dns/dns-default-2zpcz" Sep 09 02:30:47.004519 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.004425 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35546a3a-6fe9-4cff-baae-84e4a07c527a-metrics-tls\") pod \"dns-default-2zpcz\" (UID: \"35546a3a-6fe9-4cff-baae-84e4a07c527a\") " pod="openshift-dns/dns-default-2zpcz" Sep 09 02:30:47.004519 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.004457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8ksv\" (UniqueName: \"kubernetes.io/projected/35546a3a-6fe9-4cff-baae-84e4a07c527a-kube-api-access-q8ksv\") pod \"dns-default-2zpcz\" (UID: \"35546a3a-6fe9-4cff-baae-84e4a07c527a\") " pod="openshift-dns/dns-default-2zpcz" Sep 09 02:30:47.004519 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.004504 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4dj\" (UniqueName: \"kubernetes.io/projected/7aa94a6d-db29-4087-9416-92df99df14da-kube-api-access-gd4dj\") pod \"ingress-canary-z9xdr\" (UID: \"7aa94a6d-db29-4087-9416-92df99df14da\") " pod="openshift-ingress-canary/ingress-canary-z9xdr" Sep 09 02:30:47.004676 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.004659 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35546a3a-6fe9-4cff-baae-84e4a07c527a-tmp-dir\") pod \"dns-default-2zpcz\" (UID: \"35546a3a-6fe9-4cff-baae-84e4a07c527a\") " pod="openshift-dns/dns-default-2zpcz" Sep 09 02:30:47.004766 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:47.004750 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 02:30:47.004860 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:47.004846 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35546a3a-6fe9-4cff-baae-84e4a07c527a-metrics-tls podName:35546a3a-6fe9-4cff-baae-84e4a07c527a nodeName:}" failed. No retries permitted until 2026-09-09 02:30:47.504803254 +0000 UTC m=+35.635045340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35546a3a-6fe9-4cff-baae-84e4a07c527a-metrics-tls") pod "dns-default-2zpcz" (UID: "35546a3a-6fe9-4cff-baae-84e4a07c527a") : secret "dns-default-metrics-tls" not found Sep 09 02:30:47.004929 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.004910 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/35546a3a-6fe9-4cff-baae-84e4a07c527a-config-volume\") pod \"dns-default-2zpcz\" (UID: \"35546a3a-6fe9-4cff-baae-84e4a07c527a\") " pod="openshift-dns/dns-default-2zpcz" Sep 09 02:30:47.009100 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.009074 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7aa94a6d-db29-4087-9416-92df99df14da-cert\") pod \"ingress-canary-z9xdr\" (UID: \"7aa94a6d-db29-4087-9416-92df99df14da\") " pod="openshift-ingress-canary/ingress-canary-z9xdr" Sep 09 02:30:47.014745 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.014723 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8ksv\" (UniqueName: \"kubernetes.io/projected/35546a3a-6fe9-4cff-baae-84e4a07c527a-kube-api-access-q8ksv\") pod \"dns-default-2zpcz\" (UID: \"35546a3a-6fe9-4cff-baae-84e4a07c527a\") " pod="openshift-dns/dns-default-2zpcz" Sep 09 02:30:47.014971 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.014905 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd4dj\" (UniqueName: \"kubernetes.io/projected/7aa94a6d-db29-4087-9416-92df99df14da-kube-api-access-gd4dj\") pod \"ingress-canary-z9xdr\" (UID: \"7aa94a6d-db29-4087-9416-92df99df14da\") " pod="openshift-ingress-canary/ingress-canary-z9xdr" Sep 09 02:30:47.068798 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.068762 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z9xdr" Sep 09 02:30:47.303854 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.303660 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:30:47.304264 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.303664 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:30:47.304264 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.303664 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:30:47.307273 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.307256 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 02:30:47.307556 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.307539 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 02:30:47.307622 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.307578 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 02:30:47.308531 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.308516 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 02:30:47.308615 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.308568 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-shl9x\"" Sep 09 02:30:47.308676 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.308643 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rd54v\"" Sep 09 02:30:47.395384 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.395354 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z9xdr"] Sep 09 02:30:47.508810 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:47.508729 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35546a3a-6fe9-4cff-baae-84e4a07c527a-metrics-tls\") pod \"dns-default-2zpcz\" (UID: \"35546a3a-6fe9-4cff-baae-84e4a07c527a\") " pod="openshift-dns/dns-default-2zpcz" Sep 09 02:30:47.508966 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:47.508876 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 02:30:47.508966 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:47.508930 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35546a3a-6fe9-4cff-baae-84e4a07c527a-metrics-tls podName:35546a3a-6fe9-4cff-baae-84e4a07c527a nodeName:}" failed. No retries permitted until 2026-09-09 02:30:48.508916001 +0000 UTC m=+36.639158075 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35546a3a-6fe9-4cff-baae-84e4a07c527a-metrics-tls") pod "dns-default-2zpcz" (UID: "35546a3a-6fe9-4cff-baae-84e4a07c527a") : secret "dns-default-metrics-tls" not found Sep 09 02:30:48.517482 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:48.517441 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35546a3a-6fe9-4cff-baae-84e4a07c527a-metrics-tls\") pod \"dns-default-2zpcz\" (UID: \"35546a3a-6fe9-4cff-baae-84e4a07c527a\") " pod="openshift-dns/dns-default-2zpcz" Sep 09 02:30:48.518046 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:48.517573 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 02:30:48.518046 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:48.517654 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35546a3a-6fe9-4cff-baae-84e4a07c527a-metrics-tls podName:35546a3a-6fe9-4cff-baae-84e4a07c527a nodeName:}" failed. No retries permitted until 2026-09-09 02:30:50.517631459 +0000 UTC m=+38.647873553 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35546a3a-6fe9-4cff-baae-84e4a07c527a-metrics-tls") pod "dns-default-2zpcz" (UID: "35546a3a-6fe9-4cff-baae-84e4a07c527a") : secret "dns-default-metrics-tls" not found Sep 09 02:30:48.906759 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:30:48.906673 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa94a6d_db29_4087_9416_92df99df14da.slice/crio-01560bd0c4a113c1e097f56fecd6896bf4b85ca2462cf3e5fc1f7bfa75b36f98 WatchSource:0}: Error finding container 01560bd0c4a113c1e097f56fecd6896bf4b85ca2462cf3e5fc1f7bfa75b36f98: Status 404 returned error can't find the container with id 01560bd0c4a113c1e097f56fecd6896bf4b85ca2462cf3e5fc1f7bfa75b36f98 Sep 09 02:30:48.932564 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:48.932535 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:30:49.648178 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:49.648129 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z9xdr" event={"ID":"7aa94a6d-db29-4087-9416-92df99df14da","Type":"ContainerStarted","Data":"01560bd0c4a113c1e097f56fecd6896bf4b85ca2462cf3e5fc1f7bfa75b36f98"} Sep 09 02:30:49.651550 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:49.651516 2577 generic.go:356] "Generic (PLEG): container finished" podID="7bddd2af-4d25-4c54-8f2d-800934a15ab2" containerID="d8ffc3e9c1d3395f00dfcf414b0dfd2a906fb8723c6a23a7c1d8b1969007dbad" exitCode=0 Sep 09 02:30:49.651685 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:49.651579 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2ddl4" event={"ID":"7bddd2af-4d25-4c54-8f2d-800934a15ab2","Type":"ContainerDied","Data":"d8ffc3e9c1d3395f00dfcf414b0dfd2a906fb8723c6a23a7c1d8b1969007dbad"} Sep 09 02:30:50.530675 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:50.530632 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35546a3a-6fe9-4cff-baae-84e4a07c527a-metrics-tls\") pod \"dns-default-2zpcz\" (UID: \"35546a3a-6fe9-4cff-baae-84e4a07c527a\") " pod="openshift-dns/dns-default-2zpcz" Sep 09 02:30:50.530847 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:50.530772 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 02:30:50.530847 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:50.530836 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35546a3a-6fe9-4cff-baae-84e4a07c527a-metrics-tls podName:35546a3a-6fe9-4cff-baae-84e4a07c527a nodeName:}" failed. No retries permitted until 2026-09-09 02:30:54.530811336 +0000 UTC m=+42.661053410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35546a3a-6fe9-4cff-baae-84e4a07c527a-metrics-tls") pod "dns-default-2zpcz" (UID: "35546a3a-6fe9-4cff-baae-84e4a07c527a") : secret "dns-default-metrics-tls" not found Sep 09 02:30:50.657563 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:50.657529 2577 generic.go:356] "Generic (PLEG): container finished" podID="7bddd2af-4d25-4c54-8f2d-800934a15ab2" containerID="67c89dd1d99e60cd654d6d1072e8a53f1e595c4d40d19a601e29970a078852ce" exitCode=0 Sep 09 02:30:50.658058 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:50.657586 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2ddl4" event={"ID":"7bddd2af-4d25-4c54-8f2d-800934a15ab2","Type":"ContainerDied","Data":"67c89dd1d99e60cd654d6d1072e8a53f1e595c4d40d19a601e29970a078852ce"} Sep 09 02:30:51.663789 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:51.663721 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2ddl4" event={"ID":"7bddd2af-4d25-4c54-8f2d-800934a15ab2","Type":"ContainerStarted","Data":"56d837617e39c1b4d7afc36d7a808aea5096dbd0ea54d97ccc73e20fdb73e132"} Sep 09 02:30:51.694526 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:51.694483 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2ddl4" podStartSLOduration=7.321314071 podStartE2EDuration="38.694468089s" totalImagesPullingTime="31.373154018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:51.692981405 +0000 UTC m=+39.823223501" watchObservedRunningTime="2026-09-09 02:30:51.694468089 +0000 UTC m=+39.824710185" Sep 09 02:30:51.821241 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:51.821210 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:30:51.955838 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:51.955739 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:30:52.051465 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:52.051436 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:30:52.669728 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:52.669691 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z9xdr" event={"ID":"7aa94a6d-db29-4087-9416-92df99df14da","Type":"ContainerStarted","Data":"866fcdd9f9517cfd7d48f6ca195f510fe851bceada73afaa68b1078106285920"} Sep 09 02:30:52.689730 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:52.689683 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-z9xdr" podStartSLOduration=3.883626964 podStartE2EDuration="6.689667759s" totalImagesPullingTime="2.806040795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:30:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:52.688682828 +0000 UTC m=+40.818924925" watchObservedRunningTime="2026-09-09 02:30:52.689667759 +0000 UTC m=+40.819909854" Sep 09 02:30:54.562057 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:54.562015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35546a3a-6fe9-4cff-baae-84e4a07c527a-metrics-tls\") pod \"dns-default-2zpcz\" (UID: \"35546a3a-6fe9-4cff-baae-84e4a07c527a\") " pod="openshift-dns/dns-default-2zpcz" Sep 09 02:30:54.562450 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:54.562132 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 02:30:54.562450 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:30:54.562182 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35546a3a-6fe9-4cff-baae-84e4a07c527a-metrics-tls podName:35546a3a-6fe9-4cff-baae-84e4a07c527a nodeName:}" failed. No retries permitted until 2026-09-09 02:31:02.562169049 +0000 UTC m=+50.692411122 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35546a3a-6fe9-4cff-baae-84e4a07c527a-metrics-tls") pod "dns-default-2zpcz" (UID: "35546a3a-6fe9-4cff-baae-84e4a07c527a") : secret "dns-default-metrics-tls" not found Sep 09 02:30:54.568191 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:54.568165 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:30:54.665149 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:54.665121 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:30:57.146373 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:57.146340 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:30:57.215933 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:30:57.215902 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:02.612995 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:02.612934 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35546a3a-6fe9-4cff-baae-84e4a07c527a-metrics-tls\") pod \"dns-default-2zpcz\" (UID: \"35546a3a-6fe9-4cff-baae-84e4a07c527a\") " pod="openshift-dns/dns-default-2zpcz" Sep 09 02:31:02.613403 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:02.613085 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 02:31:02.613403 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:02.613148 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35546a3a-6fe9-4cff-baae-84e4a07c527a-metrics-tls podName:35546a3a-6fe9-4cff-baae-84e4a07c527a nodeName:}" failed. No retries permitted until 2026-09-09 02:31:18.61313444 +0000 UTC m=+66.743376514 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35546a3a-6fe9-4cff-baae-84e4a07c527a-metrics-tls") pod "dns-default-2zpcz" (UID: "35546a3a-6fe9-4cff-baae-84e4a07c527a") : secret "dns-default-metrics-tls" not found Sep 09 02:31:04.109555 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.109518 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-9tqqq"] Sep 09 02:31:04.111275 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.111259 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-9tqqq" Sep 09 02:31:04.116386 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.116355 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cz747"] Sep 09 02:31:04.119725 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.119650 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-scmpl\"" Sep 09 02:31:04.121581 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.121555 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cz747" Sep 09 02:31:04.125650 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.125627 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 09 02:31:04.125993 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.125978 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 09 02:31:04.126055 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.126014 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-cp5rj\"" Sep 09 02:31:04.138081 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.138057 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cz747"] Sep 09 02:31:04.138603 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.138585 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-9tqqq"] Sep 09 02:31:04.223026 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.222997 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdbpn\" (UniqueName: \"kubernetes.io/projected/1de911e0-be62-4155-819d-8b760a9b077b-kube-api-access-qdbpn\") pod \"network-check-source-685456655f-9tqqq\" (UID: \"1de911e0-be62-4155-819d-8b760a9b077b\") " pod="openshift-network-diagnostics/network-check-source-685456655f-9tqqq" Sep 09 02:31:04.223261 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.223049 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vdlj\" (UniqueName: \"kubernetes.io/projected/e03806d5-f065-4185-8899-a5b73b75befa-kube-api-access-8vdlj\") pod \"volume-data-source-validator-695c895bd6-cz747\" (UID: \"e03806d5-f065-4185-8899-a5b73b75befa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cz747" Sep 09 02:31:04.240054 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.240031 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-g849c"] Sep 09 02:31:04.242062 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.242044 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-g849c" Sep 09 02:31:04.251375 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.251347 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 09 02:31:04.251591 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.251572 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-v6g64\"" Sep 09 02:31:04.251668 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.251570 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 09 02:31:04.251668 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.251628 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 09 02:31:04.264490 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.264469 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 09 02:31:04.265066 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.265051 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 09 02:31:04.281738 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.281709 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-g849c"] Sep 09 02:31:04.323449 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.323420 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bbed170a-4fc8-4c57-a0dd-320c37ea878c-serving-cert\") pod \"console-operator-85b454874b-g849c\" (UID: \"bbed170a-4fc8-4c57-a0dd-320c37ea878c\") " pod="openshift-console-operator/console-operator-85b454874b-g849c" Sep 09 02:31:04.323632 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.323456 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdbpn\" (UniqueName: \"kubernetes.io/projected/1de911e0-be62-4155-819d-8b760a9b077b-kube-api-access-qdbpn\") pod \"network-check-source-685456655f-9tqqq\" (UID: \"1de911e0-be62-4155-819d-8b760a9b077b\") " pod="openshift-network-diagnostics/network-check-source-685456655f-9tqqq" Sep 09 02:31:04.323632 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.323476 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbed170a-4fc8-4c57-a0dd-320c37ea878c-trusted-ca\") pod \"console-operator-85b454874b-g849c\" (UID: \"bbed170a-4fc8-4c57-a0dd-320c37ea878c\") " pod="openshift-console-operator/console-operator-85b454874b-g849c" Sep 09 02:31:04.323632 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.323532 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vdlj\" (UniqueName: \"kubernetes.io/projected/e03806d5-f065-4185-8899-a5b73b75befa-kube-api-access-8vdlj\") pod \"volume-data-source-validator-695c895bd6-cz747\" (UID: \"e03806d5-f065-4185-8899-a5b73b75befa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cz747" Sep 09 02:31:04.323632 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.323547 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh8rq\" (UniqueName: \"kubernetes.io/projected/bbed170a-4fc8-4c57-a0dd-320c37ea878c-kube-api-access-hh8rq\") pod \"console-operator-85b454874b-g849c\" (UID: \"bbed170a-4fc8-4c57-a0dd-320c37ea878c\") " pod="openshift-console-operator/console-operator-85b454874b-g849c" Sep 09 02:31:04.323882 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.323684 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bbed170a-4fc8-4c57-a0dd-320c37ea878c-config\") pod \"console-operator-85b454874b-g849c\" (UID: \"bbed170a-4fc8-4c57-a0dd-320c37ea878c\") " pod="openshift-console-operator/console-operator-85b454874b-g849c" Sep 09 02:31:04.325891 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.325870 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-df69fdc44-gjkh5"] Sep 09 02:31:04.327649 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.327636 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.336105 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.336087 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 09 02:31:04.336209 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.336119 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 09 02:31:04.336209 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.336161 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dckqx\"" Sep 09 02:31:04.336209 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.336201 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 09 02:31:04.343762 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.343741 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 09 02:31:04.355337 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.355313 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdbpn\" (UniqueName: \"kubernetes.io/projected/1de911e0-be62-4155-819d-8b760a9b077b-kube-api-access-qdbpn\") pod \"network-check-source-685456655f-9tqqq\" (UID: \"1de911e0-be62-4155-819d-8b760a9b077b\") " pod="openshift-network-diagnostics/network-check-source-685456655f-9tqqq" Sep 09 02:31:04.356600 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.356580 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-df69fdc44-gjkh5"] Sep 09 02:31:04.359239 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.359221 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vdlj\" (UniqueName: \"kubernetes.io/projected/e03806d5-f065-4185-8899-a5b73b75befa-kube-api-access-8vdlj\") pod \"volume-data-source-validator-695c895bd6-cz747\" (UID: \"e03806d5-f065-4185-8899-a5b73b75befa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cz747" Sep 09 02:31:04.422660 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.422633 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-9tqqq" Sep 09 02:31:04.425917 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.424244 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bbed170a-4fc8-4c57-a0dd-320c37ea878c-serving-cert\") pod \"console-operator-85b454874b-g849c\" (UID: \"bbed170a-4fc8-4c57-a0dd-320c37ea878c\") " pod="openshift-console-operator/console-operator-85b454874b-g849c" Sep 09 02:31:04.425917 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.424289 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbed170a-4fc8-4c57-a0dd-320c37ea878c-trusted-ca\") pod \"console-operator-85b454874b-g849c\" (UID: \"bbed170a-4fc8-4c57-a0dd-320c37ea878c\") " pod="openshift-console-operator/console-operator-85b454874b-g849c" Sep 09 02:31:04.425917 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.424322 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6036480-02a7-47f9-96c0-fcbfefa2d722-trusted-ca\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.425917 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.424357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-tls\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.425917 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.424423 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh8rq\" (UniqueName: \"kubernetes.io/projected/bbed170a-4fc8-4c57-a0dd-320c37ea878c-kube-api-access-hh8rq\") pod \"console-operator-85b454874b-g849c\" (UID: \"bbed170a-4fc8-4c57-a0dd-320c37ea878c\") " pod="openshift-console-operator/console-operator-85b454874b-g849c" Sep 09 02:31:04.425917 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.424452 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e6036480-02a7-47f9-96c0-fcbfefa2d722-ca-trust-extracted\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.425917 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.424484 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-certificates\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.425917 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.424517 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-bound-sa-token\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.425917 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.424557 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e6036480-02a7-47f9-96c0-fcbfefa2d722-image-registry-private-configuration\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.425917 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.424587 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e6036480-02a7-47f9-96c0-fcbfefa2d722-installation-pull-secrets\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.425917 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.424612 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lmf8\" (UniqueName: \"kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-kube-api-access-4lmf8\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.425917 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.424645 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bbed170a-4fc8-4c57-a0dd-320c37ea878c-config\") pod \"console-operator-85b454874b-g849c\" (UID: \"bbed170a-4fc8-4c57-a0dd-320c37ea878c\") " pod="openshift-console-operator/console-operator-85b454874b-g849c" Sep 09 02:31:04.425917 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.425413 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bbed170a-4fc8-4c57-a0dd-320c37ea878c-config\") pod \"console-operator-85b454874b-g849c\" (UID: \"bbed170a-4fc8-4c57-a0dd-320c37ea878c\") " pod="openshift-console-operator/console-operator-85b454874b-g849c" Sep 09 02:31:04.427715 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.427467 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbed170a-4fc8-4c57-a0dd-320c37ea878c-trusted-ca\") pod \"console-operator-85b454874b-g849c\" (UID: \"bbed170a-4fc8-4c57-a0dd-320c37ea878c\") " pod="openshift-console-operator/console-operator-85b454874b-g849c" Sep 09 02:31:04.428112 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.428094 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bbed170a-4fc8-4c57-a0dd-320c37ea878c-serving-cert\") pod \"console-operator-85b454874b-g849c\" (UID: \"bbed170a-4fc8-4c57-a0dd-320c37ea878c\") " pod="openshift-console-operator/console-operator-85b454874b-g849c" Sep 09 02:31:04.430079 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.430060 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cz747" Sep 09 02:31:04.463932 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.463894 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh8rq\" (UniqueName: \"kubernetes.io/projected/bbed170a-4fc8-4c57-a0dd-320c37ea878c-kube-api-access-hh8rq\") pod \"console-operator-85b454874b-g849c\" (UID: \"bbed170a-4fc8-4c57-a0dd-320c37ea878c\") " pod="openshift-console-operator/console-operator-85b454874b-g849c" Sep 09 02:31:04.525374 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.525342 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6036480-02a7-47f9-96c0-fcbfefa2d722-trusted-ca\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.525559 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.525384 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-tls\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.525559 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:04.525494 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 02:31:04.525559 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:04.525509 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-df69fdc44-gjkh5: secret "image-registry-tls" not found Sep 09 02:31:04.525722 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:04.525590 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-tls podName:e6036480-02a7-47f9-96c0-fcbfefa2d722 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:05.025567953 +0000 UTC m=+53.155810041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-tls") pod "image-registry-df69fdc44-gjkh5" (UID: "e6036480-02a7-47f9-96c0-fcbfefa2d722") : secret "image-registry-tls" not found Sep 09 02:31:04.525722 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.525633 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e6036480-02a7-47f9-96c0-fcbfefa2d722-ca-trust-extracted\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.525722 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.525662 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-certificates\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.525722 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.525692 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-bound-sa-token\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.525971 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.525729 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e6036480-02a7-47f9-96c0-fcbfefa2d722-image-registry-private-configuration\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.525971 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.525756 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e6036480-02a7-47f9-96c0-fcbfefa2d722-installation-pull-secrets\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.525971 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.525781 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lmf8\" (UniqueName: \"kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-kube-api-access-4lmf8\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.526275 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.526250 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-certificates\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.526415 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.526394 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e6036480-02a7-47f9-96c0-fcbfefa2d722-ca-trust-extracted\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.526489 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.526473 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6036480-02a7-47f9-96c0-fcbfefa2d722-trusted-ca\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.528058 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.528037 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e6036480-02a7-47f9-96c0-fcbfefa2d722-image-registry-private-configuration\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.528163 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.528145 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e6036480-02a7-47f9-96c0-fcbfefa2d722-installation-pull-secrets\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.540493 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.540471 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-bound-sa-token\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.540613 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.540596 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lmf8\" (UniqueName: \"kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-kube-api-access-4lmf8\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:04.550531 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.550518 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-g849c" Sep 09 02:31:04.651039 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.651010 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:04.700688 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.700623 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-9tqqq"] Sep 09 02:31:04.703786 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:31:04.703760 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de911e0_be62_4155_819d_8b760a9b077b.slice/crio-7ca70f338ac05ca83fc2abd48ce98b9880671dfb930f588b7477a9602a05a437 WatchSource:0}: Error finding container 7ca70f338ac05ca83fc2abd48ce98b9880671dfb930f588b7477a9602a05a437: Status 404 returned error can't find the container with id 7ca70f338ac05ca83fc2abd48ce98b9880671dfb930f588b7477a9602a05a437 Sep 09 02:31:04.707021 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.706997 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cz747"] Sep 09 02:31:04.709886 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:31:04.709864 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode03806d5_f065_4185_8899_a5b73b75befa.slice/crio-46db0624852a66a6bd008d8f161e37a7b02569ea2b690b4ec7a245d142925bad WatchSource:0}: Error finding container 46db0624852a66a6bd008d8f161e37a7b02569ea2b690b4ec7a245d142925bad: Status 404 returned error can't find the container with id 46db0624852a66a6bd008d8f161e37a7b02569ea2b690b4ec7a245d142925bad Sep 09 02:31:04.746647 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.746621 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:04.828897 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:04.828867 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-g849c"] Sep 09 02:31:04.831473 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:31:04.831448 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbed170a_4fc8_4c57_a0dd_320c37ea878c.slice/crio-ce98e4c83d334dd48de8e22f19044fe388fbc3d1c038de64d8074c97898e6f78 WatchSource:0}: Error finding container ce98e4c83d334dd48de8e22f19044fe388fbc3d1c038de64d8074c97898e6f78: Status 404 returned error can't find the container with id ce98e4c83d334dd48de8e22f19044fe388fbc3d1c038de64d8074c97898e6f78 Sep 09 02:31:05.029752 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:05.029646 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-tls\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:05.029956 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:05.029843 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 02:31:05.029956 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:05.029867 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-df69fdc44-gjkh5: secret "image-registry-tls" not found Sep 09 02:31:05.029956 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:05.029940 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-tls podName:e6036480-02a7-47f9-96c0-fcbfefa2d722 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:06.029919861 +0000 UTC m=+54.160161941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-tls") pod "image-registry-df69fdc44-gjkh5" (UID: "e6036480-02a7-47f9-96c0-fcbfefa2d722") : secret "image-registry-tls" not found Sep 09 02:31:05.705272 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:05.705172 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cz747" event={"ID":"e03806d5-f065-4185-8899-a5b73b75befa","Type":"ContainerStarted","Data":"46db0624852a66a6bd008d8f161e37a7b02569ea2b690b4ec7a245d142925bad"} Sep 09 02:31:05.707148 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:05.707074 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-9tqqq" event={"ID":"1de911e0-be62-4155-819d-8b760a9b077b","Type":"ContainerStarted","Data":"7ca70f338ac05ca83fc2abd48ce98b9880671dfb930f588b7477a9602a05a437"} Sep 09 02:31:05.709051 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:05.708980 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-g849c" event={"ID":"bbed170a-4fc8-4c57-a0dd-320c37ea878c","Type":"ContainerStarted","Data":"ce98e4c83d334dd48de8e22f19044fe388fbc3d1c038de64d8074c97898e6f78"} Sep 09 02:31:06.038185 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:06.038095 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-tls\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:06.038353 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:06.038263 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 02:31:06.038353 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:06.038285 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-df69fdc44-gjkh5: secret "image-registry-tls" not found Sep 09 02:31:06.038353 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:06.038351 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-tls podName:e6036480-02a7-47f9-96c0-fcbfefa2d722 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:08.03833375 +0000 UTC m=+56.168575841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-tls") pod "image-registry-df69fdc44-gjkh5" (UID: "e6036480-02a7-47f9-96c0-fcbfefa2d722") : secret "image-registry-tls" not found Sep 09 02:31:06.581624 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:06.581589 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:06.655275 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:06.655231 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:06.713520 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:06.713477 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cz747" event={"ID":"e03806d5-f065-4185-8899-a5b73b75befa","Type":"ContainerStarted","Data":"a89899327ce0e8a1f50ced5445cd7c8f967854a18c46610398e14ef13af0a643"} Sep 09 02:31:06.738656 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:06.738581 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cz747" podStartSLOduration=1.6240437509999999 podStartE2EDuration="2.738560161s" totalImagesPullingTime="1.11451641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:06.737757613 +0000 UTC m=+54.867999712" watchObservedRunningTime="2026-09-09 02:31:06.738560161 +0000 UTC m=+54.868802259" Sep 09 02:31:06.870980 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:06.870909 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:06.939227 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:06.939197 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:07.316465 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:07.316401 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-j5465"] Sep 09 02:31:07.318395 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:07.318374 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-j5465" Sep 09 02:31:07.321066 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:07.321042 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 09 02:31:07.321395 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:07.321379 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-rsbbl\"" Sep 09 02:31:07.322236 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:07.322217 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 09 02:31:07.333134 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:07.333114 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-j5465"] Sep 09 02:31:07.449300 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:07.449270 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3f5283dd-ae41-4c1a-a1d3-84285f55e969-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-j5465\" (UID: \"3f5283dd-ae41-4c1a-a1d3-84285f55e969\") " pod="openshift-network-console/networking-console-plugin-79977f9844-j5465" Sep 09 02:31:07.449505 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:07.449389 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3f5283dd-ae41-4c1a-a1d3-84285f55e969-nginx-conf\") pod \"networking-console-plugin-79977f9844-j5465\" (UID: \"3f5283dd-ae41-4c1a-a1d3-84285f55e969\") " pod="openshift-network-console/networking-console-plugin-79977f9844-j5465" Sep 09 02:31:07.550423 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:07.550383 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3f5283dd-ae41-4c1a-a1d3-84285f55e969-nginx-conf\") pod \"networking-console-plugin-79977f9844-j5465\" (UID: \"3f5283dd-ae41-4c1a-a1d3-84285f55e969\") " pod="openshift-network-console/networking-console-plugin-79977f9844-j5465" Sep 09 02:31:07.550592 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:07.550492 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3f5283dd-ae41-4c1a-a1d3-84285f55e969-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-j5465\" (UID: \"3f5283dd-ae41-4c1a-a1d3-84285f55e969\") " pod="openshift-network-console/networking-console-plugin-79977f9844-j5465" Sep 09 02:31:07.550639 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:07.550607 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 02:31:07.550694 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:07.550676 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f5283dd-ae41-4c1a-a1d3-84285f55e969-networking-console-plugin-cert podName:3f5283dd-ae41-4c1a-a1d3-84285f55e969 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:08.050658369 +0000 UTC m=+56.180900443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3f5283dd-ae41-4c1a-a1d3-84285f55e969-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-j5465" (UID: "3f5283dd-ae41-4c1a-a1d3-84285f55e969") : secret "networking-console-plugin-cert" not found Sep 09 02:31:07.551195 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:07.551176 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3f5283dd-ae41-4c1a-a1d3-84285f55e969-nginx-conf\") pod \"networking-console-plugin-79977f9844-j5465\" (UID: \"3f5283dd-ae41-4c1a-a1d3-84285f55e969\") " pod="openshift-network-console/networking-console-plugin-79977f9844-j5465" Sep 09 02:31:08.054405 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:08.054373 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-tls\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:08.054770 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:08.054425 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3f5283dd-ae41-4c1a-a1d3-84285f55e969-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-j5465\" (UID: \"3f5283dd-ae41-4c1a-a1d3-84285f55e969\") " pod="openshift-network-console/networking-console-plugin-79977f9844-j5465" Sep 09 02:31:08.054770 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:08.054509 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 02:31:08.054770 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:08.054528 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 02:31:08.054770 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:08.054548 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-df69fdc44-gjkh5: secret "image-registry-tls" not found Sep 09 02:31:08.054770 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:08.054557 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f5283dd-ae41-4c1a-a1d3-84285f55e969-networking-console-plugin-cert podName:3f5283dd-ae41-4c1a-a1d3-84285f55e969 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:09.054544947 +0000 UTC m=+57.184787026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3f5283dd-ae41-4c1a-a1d3-84285f55e969-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-j5465" (UID: "3f5283dd-ae41-4c1a-a1d3-84285f55e969") : secret "networking-console-plugin-cert" not found Sep 09 02:31:08.054770 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:08.054600 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-tls podName:e6036480-02a7-47f9-96c0-fcbfefa2d722 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:12.054582939 +0000 UTC m=+60.184825021 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-tls") pod "image-registry-df69fdc44-gjkh5" (UID: "e6036480-02a7-47f9-96c0-fcbfefa2d722") : secret "image-registry-tls" not found Sep 09 02:31:08.720245 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:08.720204 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-9tqqq" event={"ID":"1de911e0-be62-4155-819d-8b760a9b077b","Type":"ContainerStarted","Data":"378df6b388a357432fa9841281d4a429e0b374454db650a9348747b661da1955"} Sep 09 02:31:08.721898 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:08.721878 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/0.log" Sep 09 02:31:08.722012 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:08.721913 2577 generic.go:356] "Generic (PLEG): container finished" podID="bbed170a-4fc8-4c57-a0dd-320c37ea878c" containerID="0cf4cbe8a2c3bcbd1d5b549b089020c154b0fa9ac94f74853958bbfd3d42bc6a" exitCode=255 Sep 09 02:31:08.722012 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:08.721936 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-g849c" event={"ID":"bbed170a-4fc8-4c57-a0dd-320c37ea878c","Type":"ContainerDied","Data":"0cf4cbe8a2c3bcbd1d5b549b089020c154b0fa9ac94f74853958bbfd3d42bc6a"} Sep 09 02:31:08.722163 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:08.722148 2577 scope.go:117] "RemoveContainer" containerID="0cf4cbe8a2c3bcbd1d5b549b089020c154b0fa9ac94f74853958bbfd3d42bc6a" Sep 09 02:31:08.774490 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:08.774444 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-9tqqq" podStartSLOduration=1.631509006 podStartE2EDuration="4.774425777s" totalImagesPullingTime="3.142916771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:08.745937619 +0000 UTC m=+56.876179716" watchObservedRunningTime="2026-09-09 02:31:08.774425777 +0000 UTC m=+56.904667873" Sep 09 02:31:08.945561 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:08.945523 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:09.006482 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.006408 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mq79l"] Sep 09 02:31:09.008506 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.008491 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:09.016482 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.016460 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 09 02:31:09.017042 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.017017 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 09 02:31:09.017042 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.017035 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-68fdc\"" Sep 09 02:31:09.017182 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.017080 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 09 02:31:09.017182 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.017154 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 09 02:31:09.032227 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.032199 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mq79l"] Sep 09 02:31:09.063306 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.063275 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3f5283dd-ae41-4c1a-a1d3-84285f55e969-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-j5465\" (UID: \"3f5283dd-ae41-4c1a-a1d3-84285f55e969\") " pod="openshift-network-console/networking-console-plugin-79977f9844-j5465" Sep 09 02:31:09.063758 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:09.063476 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 02:31:09.063758 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:09.063549 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f5283dd-ae41-4c1a-a1d3-84285f55e969-networking-console-plugin-cert podName:3f5283dd-ae41-4c1a-a1d3-84285f55e969 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:11.063528656 +0000 UTC m=+59.193770750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3f5283dd-ae41-4c1a-a1d3-84285f55e969-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-j5465" (UID: "3f5283dd-ae41-4c1a-a1d3-84285f55e969") : secret "networking-console-plugin-cert" not found Sep 09 02:31:09.069450 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.069422 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:09.163645 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.163610 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5a72e3f-0e47-4305-b6f4-b9f294b90913-crio-socket\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:09.163832 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.163665 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5a72e3f-0e47-4305-b6f4-b9f294b90913-data-volume\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:09.163832 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.163719 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5a72e3f-0e47-4305-b6f4-b9f294b90913-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:09.163832 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.163805 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcgz6\" (UniqueName: \"kubernetes.io/projected/b5a72e3f-0e47-4305-b6f4-b9f294b90913-kube-api-access-wcgz6\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:09.163933 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.163910 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5a72e3f-0e47-4305-b6f4-b9f294b90913-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:09.199939 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.199907 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:09.264682 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.264604 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5a72e3f-0e47-4305-b6f4-b9f294b90913-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:09.264682 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.264641 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5a72e3f-0e47-4305-b6f4-b9f294b90913-crio-socket\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:09.264682 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.264669 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5a72e3f-0e47-4305-b6f4-b9f294b90913-data-volume\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:09.264682 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.264685 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5a72e3f-0e47-4305-b6f4-b9f294b90913-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:09.264953 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:09.264740 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 02:31:09.264953 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:09.264836 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 02:31:09.264953 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.264744 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5a72e3f-0e47-4305-b6f4-b9f294b90913-crio-socket\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:09.264953 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.264745 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcgz6\" (UniqueName: \"kubernetes.io/projected/b5a72e3f-0e47-4305-b6f4-b9f294b90913-kube-api-access-wcgz6\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:09.264953 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:09.264841 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5a72e3f-0e47-4305-b6f4-b9f294b90913-kube-rbac-proxy-cm podName:b5a72e3f-0e47-4305-b6f4-b9f294b90913 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:09.764801296 +0000 UTC m=+57.895043396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b5a72e3f-0e47-4305-b6f4-b9f294b90913-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mq79l" (UID: "b5a72e3f-0e47-4305-b6f4-b9f294b90913") : configmap "kube-rbac-proxy" not found Sep 09 02:31:09.264953 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:09.264928 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5a72e3f-0e47-4305-b6f4-b9f294b90913-insights-runtime-extractor-tls podName:b5a72e3f-0e47-4305-b6f4-b9f294b90913 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:09.764900217 +0000 UTC m=+57.895142305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b5a72e3f-0e47-4305-b6f4-b9f294b90913-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mq79l" (UID: "b5a72e3f-0e47-4305-b6f4-b9f294b90913") : secret "insights-runtime-extractor-tls" not found Sep 09 02:31:09.265227 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.265032 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5a72e3f-0e47-4305-b6f4-b9f294b90913-data-volume\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:09.276004 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.275981 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcgz6\" (UniqueName: \"kubernetes.io/projected/b5a72e3f-0e47-4305-b6f4-b9f294b90913-kube-api-access-wcgz6\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:09.300510 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.300483 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:09.614267 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.614184 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:09.722157 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.722126 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:09.725446 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.725424 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/1.log" Sep 09 02:31:09.725858 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.725842 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/0.log" Sep 09 02:31:09.725960 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.725880 2577 generic.go:356] "Generic (PLEG): container finished" podID="bbed170a-4fc8-4c57-a0dd-320c37ea878c" containerID="f7f38850b1c6458600ac951b99b13159a1f7765a4df41bad8966999b310eddb6" exitCode=255 Sep 09 02:31:09.725960 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.725927 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-g849c" event={"ID":"bbed170a-4fc8-4c57-a0dd-320c37ea878c","Type":"ContainerDied","Data":"f7f38850b1c6458600ac951b99b13159a1f7765a4df41bad8966999b310eddb6"} Sep 09 02:31:09.726047 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.725982 2577 scope.go:117] "RemoveContainer" containerID="0cf4cbe8a2c3bcbd1d5b549b089020c154b0fa9ac94f74853958bbfd3d42bc6a" Sep 09 02:31:09.726215 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.726188 2577 scope.go:117] "RemoveContainer" containerID="f7f38850b1c6458600ac951b99b13159a1f7765a4df41bad8966999b310eddb6" Sep 09 02:31:09.726418 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:09.726399 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-g849c_openshift-console-operator(bbed170a-4fc8-4c57-a0dd-320c37ea878c)\"" pod="openshift-console-operator/console-operator-85b454874b-g849c" podUID="bbed170a-4fc8-4c57-a0dd-320c37ea878c" Sep 09 02:31:09.769436 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.769407 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5a72e3f-0e47-4305-b6f4-b9f294b90913-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:09.769585 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:09.769541 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 02:31:09.769648 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:09.769593 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5a72e3f-0e47-4305-b6f4-b9f294b90913-insights-runtime-extractor-tls podName:b5a72e3f-0e47-4305-b6f4-b9f294b90913 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:10.769578341 +0000 UTC m=+58.899820415 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b5a72e3f-0e47-4305-b6f4-b9f294b90913-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mq79l" (UID: "b5a72e3f-0e47-4305-b6f4-b9f294b90913") : secret "insights-runtime-extractor-tls" not found Sep 09 02:31:09.769648 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.769542 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5a72e3f-0e47-4305-b6f4-b9f294b90913-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:09.769725 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:09.769600 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 02:31:09.769725 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:09.769718 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5a72e3f-0e47-4305-b6f4-b9f294b90913-kube-rbac-proxy-cm podName:b5a72e3f-0e47-4305-b6f4-b9f294b90913 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:10.769705266 +0000 UTC m=+58.899947339 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b5a72e3f-0e47-4305-b6f4-b9f294b90913-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mq79l" (UID: "b5a72e3f-0e47-4305-b6f4-b9f294b90913") : configmap "kube-rbac-proxy" not found Sep 09 02:31:09.982091 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:09.982054 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:10.007032 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:10.006998 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mf8qc"] Sep 09 02:31:10.009714 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:10.009695 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mf8qc" Sep 09 02:31:10.028907 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:10.028887 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-f52df\"" Sep 09 02:31:10.031674 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:10.031658 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 09 02:31:10.042627 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:10.042603 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 09 02:31:10.047080 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:10.047057 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mf8qc"] Sep 09 02:31:10.138502 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:10.138470 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:10.173826 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:10.173786 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7xnz\" (UniqueName: \"kubernetes.io/projected/d948a89d-7114-420b-a168-c6fdc6ba64e1-kube-api-access-t7xnz\") pod \"migrator-fdbb8fdff-mf8qc\" (UID: \"d948a89d-7114-420b-a168-c6fdc6ba64e1\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mf8qc" Sep 09 02:31:10.238859 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:10.238771 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:10.274401 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:10.274375 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7xnz\" (UniqueName: \"kubernetes.io/projected/d948a89d-7114-420b-a168-c6fdc6ba64e1-kube-api-access-t7xnz\") pod \"migrator-fdbb8fdff-mf8qc\" (UID: \"d948a89d-7114-420b-a168-c6fdc6ba64e1\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mf8qc" Sep 09 02:31:10.284582 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:10.284564 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7xnz\" (UniqueName: \"kubernetes.io/projected/d948a89d-7114-420b-a168-c6fdc6ba64e1-kube-api-access-t7xnz\") pod \"migrator-fdbb8fdff-mf8qc\" (UID: \"d948a89d-7114-420b-a168-c6fdc6ba64e1\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mf8qc" Sep 09 02:31:10.318198 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:10.318173 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mf8qc" Sep 09 02:31:10.601661 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:10.601589 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mf8qc"] Sep 09 02:31:10.604760 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:31:10.604735 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd948a89d_7114_420b_a168_c6fdc6ba64e1.slice/crio-2e192c5eb33413d13c006564bcd275fa8a649adaa9419c7ef89167ac664d4f36 WatchSource:0}: Error finding container 2e192c5eb33413d13c006564bcd275fa8a649adaa9419c7ef89167ac664d4f36: Status 404 returned error can't find the container with id 2e192c5eb33413d13c006564bcd275fa8a649adaa9419c7ef89167ac664d4f36 Sep 09 02:31:10.729595 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:10.729563 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mf8qc" event={"ID":"d948a89d-7114-420b-a168-c6fdc6ba64e1","Type":"ContainerStarted","Data":"2e192c5eb33413d13c006564bcd275fa8a649adaa9419c7ef89167ac664d4f36"} Sep 09 02:31:10.730917 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:10.730899 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/1.log" Sep 09 02:31:10.731208 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:10.731186 2577 scope.go:117] "RemoveContainer" containerID="f7f38850b1c6458600ac951b99b13159a1f7765a4df41bad8966999b310eddb6" Sep 09 02:31:10.731365 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:10.731348 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-g849c_openshift-console-operator(bbed170a-4fc8-4c57-a0dd-320c37ea878c)\"" pod="openshift-console-operator/console-operator-85b454874b-g849c" podUID="bbed170a-4fc8-4c57-a0dd-320c37ea878c" Sep 09 02:31:10.778899 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:10.778869 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5a72e3f-0e47-4305-b6f4-b9f294b90913-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:10.779060 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:10.778923 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5a72e3f-0e47-4305-b6f4-b9f294b90913-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:10.779060 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:10.778997 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 02:31:10.779131 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:10.779062 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5a72e3f-0e47-4305-b6f4-b9f294b90913-kube-rbac-proxy-cm podName:b5a72e3f-0e47-4305-b6f4-b9f294b90913 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:12.779044878 +0000 UTC m=+60.909286956 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b5a72e3f-0e47-4305-b6f4-b9f294b90913-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mq79l" (UID: "b5a72e3f-0e47-4305-b6f4-b9f294b90913") : configmap "kube-rbac-proxy" not found Sep 09 02:31:10.779131 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:10.779068 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 02:31:10.779131 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:10.779112 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5a72e3f-0e47-4305-b6f4-b9f294b90913-insights-runtime-extractor-tls podName:b5a72e3f-0e47-4305-b6f4-b9f294b90913 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:12.779097097 +0000 UTC m=+60.909339174 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b5a72e3f-0e47-4305-b6f4-b9f294b90913-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mq79l" (UID: "b5a72e3f-0e47-4305-b6f4-b9f294b90913") : secret "insights-runtime-extractor-tls" not found Sep 09 02:31:11.054348 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:11.054323 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wr95f_40631b35-00b4-4a4b-ab2e-f769434da1f1/dns-node-resolver/0.log" Sep 09 02:31:11.081672 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:11.081636 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3f5283dd-ae41-4c1a-a1d3-84285f55e969-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-j5465\" (UID: \"3f5283dd-ae41-4c1a-a1d3-84285f55e969\") " pod="openshift-network-console/networking-console-plugin-79977f9844-j5465" Sep 09 02:31:11.081856 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:11.081799 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 02:31:11.081918 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:11.081901 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f5283dd-ae41-4c1a-a1d3-84285f55e969-networking-console-plugin-cert podName:3f5283dd-ae41-4c1a-a1d3-84285f55e969 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:15.081879053 +0000 UTC m=+63.212121129 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3f5283dd-ae41-4c1a-a1d3-84285f55e969-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-j5465" (UID: "3f5283dd-ae41-4c1a-a1d3-84285f55e969") : secret "networking-console-plugin-cert" not found Sep 09 02:31:11.571485 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:11.571449 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:11.657931 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:11.657913 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qfvvm_2f959dac-774e-473c-b318-7cec8464e980/node-ca/0.log" Sep 09 02:31:11.671713 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:11.671690 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:11.736894 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:11.736862 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mf8qc" event={"ID":"d948a89d-7114-420b-a168-c6fdc6ba64e1","Type":"ContainerStarted","Data":"d7949b4b7007ae9f75d00f36c2946d75ecc57446fb4279163aaabed4aa28a415"} Sep 09 02:31:11.737033 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:11.736912 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mf8qc" event={"ID":"d948a89d-7114-420b-a168-c6fdc6ba64e1","Type":"ContainerStarted","Data":"696a524d2eebb33ce60fac46459d3d37a5cb2c16843ed4890024442d482b5255"} Sep 09 02:31:11.755917 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:11.755872 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mf8qc" podStartSLOduration=1.8161479040000001 podStartE2EDuration="2.755856288s" totalImagesPullingTime="939.708384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:11.755441061 +0000 UTC m=+59.885683160" watchObservedRunningTime="2026-09-09 02:31:11.755856288 +0000 UTC m=+59.886098385" Sep 09 02:31:11.856219 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:11.856131 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:11.925510 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:11.925480 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:12.089542 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:12.089511 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-tls\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:12.089693 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:12.089659 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 02:31:12.089693 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:12.089676 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-df69fdc44-gjkh5: secret "image-registry-tls" not found Sep 09 02:31:12.089761 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:12.089727 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-tls podName:e6036480-02a7-47f9-96c0-fcbfefa2d722 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:20.089713778 +0000 UTC m=+68.219955852 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-tls") pod "image-registry-df69fdc44-gjkh5" (UID: "e6036480-02a7-47f9-96c0-fcbfefa2d722") : secret "image-registry-tls" not found Sep 09 02:31:12.488008 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:12.487975 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:12.795790 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:12.795711 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5a72e3f-0e47-4305-b6f4-b9f294b90913-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:12.795790 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:12.795760 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5a72e3f-0e47-4305-b6f4-b9f294b90913-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:12.796196 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:12.795842 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 02:31:12.796196 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:12.795903 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5a72e3f-0e47-4305-b6f4-b9f294b90913-kube-rbac-proxy-cm podName:b5a72e3f-0e47-4305-b6f4-b9f294b90913 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:16.795888372 +0000 UTC m=+64.926130450 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b5a72e3f-0e47-4305-b6f4-b9f294b90913-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mq79l" (UID: "b5a72e3f-0e47-4305-b6f4-b9f294b90913") : configmap "kube-rbac-proxy" not found Sep 09 02:31:12.796196 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:12.795910 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 02:31:12.796196 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:12.795953 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5a72e3f-0e47-4305-b6f4-b9f294b90913-insights-runtime-extractor-tls podName:b5a72e3f-0e47-4305-b6f4-b9f294b90913 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:16.795938853 +0000 UTC m=+64.926180928 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b5a72e3f-0e47-4305-b6f4-b9f294b90913-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mq79l" (UID: "b5a72e3f-0e47-4305-b6f4-b9f294b90913") : secret "insights-runtime-extractor-tls" not found Sep 09 02:31:12.856136 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:12.856109 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z9xdr_7aa94a6d-db29-4087-9416-92df99df14da/serve-healthcheck-canary/0.log" Sep 09 02:31:13.642300 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:13.642274 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t8g9x" Sep 09 02:31:14.550824 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:14.550777 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-g849c" Sep 09 02:31:14.551277 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:14.550845 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-g849c" Sep 09 02:31:14.551277 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:14.551165 2577 scope.go:117] "RemoveContainer" containerID="f7f38850b1c6458600ac951b99b13159a1f7765a4df41bad8966999b310eddb6" Sep 09 02:31:14.551363 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:14.551342 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-g849c_openshift-console-operator(bbed170a-4fc8-4c57-a0dd-320c37ea878c)\"" pod="openshift-console-operator/console-operator-85b454874b-g849c" podUID="bbed170a-4fc8-4c57-a0dd-320c37ea878c" Sep 09 02:31:15.113672 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:15.113635 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3f5283dd-ae41-4c1a-a1d3-84285f55e969-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-j5465\" (UID: \"3f5283dd-ae41-4c1a-a1d3-84285f55e969\") " pod="openshift-network-console/networking-console-plugin-79977f9844-j5465" Sep 09 02:31:15.113883 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:15.113740 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 02:31:15.113883 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:15.113800 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f5283dd-ae41-4c1a-a1d3-84285f55e969-networking-console-plugin-cert podName:3f5283dd-ae41-4c1a-a1d3-84285f55e969 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:23.113786256 +0000 UTC m=+71.244028335 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3f5283dd-ae41-4c1a-a1d3-84285f55e969-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-j5465" (UID: "3f5283dd-ae41-4c1a-a1d3-84285f55e969") : secret "networking-console-plugin-cert" not found Sep 09 02:31:15.549540 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:15.549505 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:15.658478 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:15.658446 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:15.767865 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:15.767837 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:15.884338 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:15.884313 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:15.995105 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:15.995075 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:16.100674 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:16.100648 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:16.429536 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:16.429504 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:16.627716 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:16.627686 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:16.827936 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:16.827839 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5a72e3f-0e47-4305-b6f4-b9f294b90913-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:16.827936 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:16.827905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5a72e3f-0e47-4305-b6f4-b9f294b90913-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:16.828370 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:16.827960 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 02:31:16.828370 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:16.828047 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5a72e3f-0e47-4305-b6f4-b9f294b90913-kube-rbac-proxy-cm podName:b5a72e3f-0e47-4305-b6f4-b9f294b90913 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:24.828027893 +0000 UTC m=+72.958269982 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b5a72e3f-0e47-4305-b6f4-b9f294b90913-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mq79l" (UID: "b5a72e3f-0e47-4305-b6f4-b9f294b90913") : configmap "kube-rbac-proxy" not found Sep 09 02:31:16.828370 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:16.828055 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 02:31:16.828370 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:16.828093 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5a72e3f-0e47-4305-b6f4-b9f294b90913-insights-runtime-extractor-tls podName:b5a72e3f-0e47-4305-b6f4-b9f294b90913 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:24.828082224 +0000 UTC m=+72.958324302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b5a72e3f-0e47-4305-b6f4-b9f294b90913-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mq79l" (UID: "b5a72e3f-0e47-4305-b6f4-b9f294b90913") : secret "insights-runtime-extractor-tls" not found Sep 09 02:31:17.030230 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:17.030193 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret\") pod \"global-pull-secret-syncer-4fb84\" (UID: \"e0fea0ae-0424-4574-b264-04ea4bcdea35\") " pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:31:17.033392 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:17.033376 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 02:31:17.043408 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:17.043385 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0fea0ae-0424-4574-b264-04ea4bcdea35-original-pull-secret\") pod \"global-pull-secret-syncer-4fb84\" (UID: \"e0fea0ae-0424-4574-b264-04ea4bcdea35\") " pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:31:17.131640 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:17.131610 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs\") pod \"network-metrics-daemon-n7lh6\" (UID: \"59a52c60-f713-4a2f-8de1-5d7a2d43ac41\") " pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:31:17.134612 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:17.134595 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 02:31:17.142056 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:17.142035 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 02:31:17.142122 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:17.142107 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs podName:59a52c60-f713-4a2f-8de1-5d7a2d43ac41 nodeName:}" failed. No retries permitted until 2026-09-09 02:32:21.142085582 +0000 UTC m=+129.272327675 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs") pod "network-metrics-daemon-n7lh6" (UID: "59a52c60-f713-4a2f-8de1-5d7a2d43ac41") : secret "metrics-daemon-secret" not found Sep 09 02:31:17.232611 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:17.232579 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlt7n\" (UniqueName: \"kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n\") pod \"network-check-target-zr2ql\" (UID: \"6e6ad2e4-55a9-4509-99b4-30958e61e3a6\") " pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:31:17.235048 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:17.235029 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlt7n\" (UniqueName: \"kubernetes.io/projected/6e6ad2e4-55a9-4509-99b4-30958e61e3a6-kube-api-access-dlt7n\") pod \"network-check-target-zr2ql\" (UID: \"6e6ad2e4-55a9-4509-99b4-30958e61e3a6\") " pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:31:17.313392 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:17.313353 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fb84" Sep 09 02:31:17.323443 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:17.323406 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rd54v\"" Sep 09 02:31:17.330842 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:17.330804 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:31:17.448373 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:17.448317 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:17.599346 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:17.599320 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4fb84"] Sep 09 02:31:17.601662 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:31:17.601636 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0fea0ae_0424_4574_b264_04ea4bcdea35.slice/crio-e7da522d4b4d91528c139ef034821818d6e31a6b502b65584614a4700540a88e WatchSource:0}: Error finding container e7da522d4b4d91528c139ef034821818d6e31a6b502b65584614a4700540a88e: Status 404 returned error can't find the container with id e7da522d4b4d91528c139ef034821818d6e31a6b502b65584614a4700540a88e Sep 09 02:31:17.619872 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:17.619849 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zr2ql"] Sep 09 02:31:17.622541 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:31:17.622517 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e6ad2e4_55a9_4509_99b4_30958e61e3a6.slice/crio-d140c96cd0f1fcfd087d79b4db2fe204ca5b4bf13aafde8ca9ce4a2b0c2415d2 WatchSource:0}: Error finding container d140c96cd0f1fcfd087d79b4db2fe204ca5b4bf13aafde8ca9ce4a2b0c2415d2: Status 404 returned error can't find the container with id d140c96cd0f1fcfd087d79b4db2fe204ca5b4bf13aafde8ca9ce4a2b0c2415d2 Sep 09 02:31:17.630230 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:17.630211 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:17.755954 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:17.755925 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4fb84" event={"ID":"e0fea0ae-0424-4574-b264-04ea4bcdea35","Type":"ContainerStarted","Data":"e7da522d4b4d91528c139ef034821818d6e31a6b502b65584614a4700540a88e"} Sep 09 02:31:17.757362 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:17.757341 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zr2ql" event={"ID":"6e6ad2e4-55a9-4509-99b4-30958e61e3a6","Type":"ContainerStarted","Data":"b67d9ce53d94f43870cb2711d5be7cc291606e18eaa76103b190105838840f29"} Sep 09 02:31:17.757470 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:17.757375 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zr2ql" event={"ID":"6e6ad2e4-55a9-4509-99b4-30958e61e3a6","Type":"ContainerStarted","Data":"d140c96cd0f1fcfd087d79b4db2fe204ca5b4bf13aafde8ca9ce4a2b0c2415d2"} Sep 09 02:31:17.757509 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:17.757478 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:31:18.257234 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:18.257201 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:18.643464 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:18.643427 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35546a3a-6fe9-4cff-baae-84e4a07c527a-metrics-tls\") pod \"dns-default-2zpcz\" (UID: \"35546a3a-6fe9-4cff-baae-84e4a07c527a\") " pod="openshift-dns/dns-default-2zpcz" Sep 09 02:31:18.646257 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:18.646224 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35546a3a-6fe9-4cff-baae-84e4a07c527a-metrics-tls\") pod \"dns-default-2zpcz\" (UID: \"35546a3a-6fe9-4cff-baae-84e4a07c527a\") " pod="openshift-dns/dns-default-2zpcz" Sep 09 02:31:18.647222 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:18.647181 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:18.854585 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:18.854556 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-99mnl\"" Sep 09 02:31:18.862282 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:18.862258 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2zpcz" Sep 09 02:31:19.018158 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:19.018123 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:19.146421 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:19.146369 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zr2ql" podStartSLOduration=66.146345478 podStartE2EDuration="1m6.146345478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:17.784808146 +0000 UTC m=+65.915050243" watchObservedRunningTime="2026-09-09 02:31:19.146345478 +0000 UTC m=+67.276587602" Sep 09 02:31:19.146698 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:19.146586 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2zpcz"] Sep 09 02:31:19.148714 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:31:19.148687 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35546a3a_6fe9_4cff_baae_84e4a07c527a.slice/crio-85b806f70ab5a81fb63731b1960e4b455944912f45feb2b403e7b926f3549acb WatchSource:0}: Error finding container 85b806f70ab5a81fb63731b1960e4b455944912f45feb2b403e7b926f3549acb: Status 404 returned error can't find the container with id 85b806f70ab5a81fb63731b1960e4b455944912f45feb2b403e7b926f3549acb Sep 09 02:31:19.240745 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:19.240644 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:19.341271 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:19.341238 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:19.681718 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:19.681646 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:19.765464 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:19.765426 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2zpcz" event={"ID":"35546a3a-6fe9-4cff-baae-84e4a07c527a","Type":"ContainerStarted","Data":"85b806f70ab5a81fb63731b1960e4b455944912f45feb2b403e7b926f3549acb"} Sep 09 02:31:19.838024 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:19.837990 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:20.155318 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:20.155281 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-tls\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:20.158342 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:20.158312 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-tls\") pod \"image-registry-df69fdc44-gjkh5\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:20.240767 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:20.240736 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dckqx\"" Sep 09 02:31:20.246809 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:20.246753 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:20.301168 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:20.301132 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:20.424109 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:20.424009 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:20.523779 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:20.523737 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:21.017299 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:21.017263 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:21.420024 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:21.419999 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:21.618339 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:21.618305 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:21.840302 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:21.840211 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:22.020182 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:22.020150 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:22.132611 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:22.132583 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:22.623701 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:22.623669 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-df69fdc44-gjkh5"] Sep 09 02:31:22.627491 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:31:22.627458 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6036480_02a7_47f9_96c0_fcbfefa2d722.slice/crio-6ac490240db4122c866ab34c453eac56a880f2ff1a60b1410bc52ed7f575a415 WatchSource:0}: Error finding container 6ac490240db4122c866ab34c453eac56a880f2ff1a60b1410bc52ed7f575a415: Status 404 returned error can't find the container with id 6ac490240db4122c866ab34c453eac56a880f2ff1a60b1410bc52ed7f575a415 Sep 09 02:31:22.778091 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:22.778057 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4fb84" event={"ID":"e0fea0ae-0424-4574-b264-04ea4bcdea35","Type":"ContainerStarted","Data":"4c143be42f8fddc3acaa2937b1a9e3ea3a3be627204e551896e7e3c171842595"} Sep 09 02:31:22.780093 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:22.780063 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2zpcz" event={"ID":"35546a3a-6fe9-4cff-baae-84e4a07c527a","Type":"ContainerStarted","Data":"42b53c96e8f1bd47a1124d221a751b7900473d46d9ad3caac8bcc82bfe8d10e5"} Sep 09 02:31:22.780230 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:22.780110 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2zpcz" event={"ID":"35546a3a-6fe9-4cff-baae-84e4a07c527a","Type":"ContainerStarted","Data":"16005aa5b6dc32009f202be4b5768eeb7ce907946e00b5a881dd8e441791100a"} Sep 09 02:31:22.780230 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:22.780159 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2zpcz" Sep 09 02:31:22.781724 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:22.781703 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" event={"ID":"e6036480-02a7-47f9-96c0-fcbfefa2d722","Type":"ContainerStarted","Data":"a637bca61b5779a5103b651f2c1000d784df76ac626b27325195fe6c5a86cb95"} Sep 09 02:31:22.781863 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:22.781746 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" event={"ID":"e6036480-02a7-47f9-96c0-fcbfefa2d722","Type":"ContainerStarted","Data":"6ac490240db4122c866ab34c453eac56a880f2ff1a60b1410bc52ed7f575a415"} Sep 09 02:31:22.781863 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:22.781770 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:22.808943 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:22.808878 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4fb84" podStartSLOduration=65.063299607 podStartE2EDuration="1m9.808856655s" totalImagesPullingTime="4.745557048s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:22.808316754 +0000 UTC m=+70.938558851" watchObservedRunningTime="2026-09-09 02:31:22.808856655 +0000 UTC m=+70.939098754" Sep 09 02:31:22.836272 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:22.836225 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" podStartSLOduration=18.836210746 podStartE2EDuration="18.836210746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:22.834421633 +0000 UTC m=+70.964663730" watchObservedRunningTime="2026-09-09 02:31:22.836210746 +0000 UTC m=+70.966452842" Sep 09 02:31:22.840022 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:22.839996 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:22.859975 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:22.859921 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2zpcz" podStartSLOduration=33.658513416 podStartE2EDuration="36.859905117s" totalImagesPullingTime="3.201391701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:30:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:22.857527224 +0000 UTC m=+70.987769319" watchObservedRunningTime="2026-09-09 02:31:22.859905117 +0000 UTC m=+70.990147213" Sep 09 02:31:23.025786 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:23.025714 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:23.179357 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:23.179318 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3f5283dd-ae41-4c1a-a1d3-84285f55e969-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-j5465\" (UID: \"3f5283dd-ae41-4c1a-a1d3-84285f55e969\") " pod="openshift-network-console/networking-console-plugin-79977f9844-j5465" Sep 09 02:31:23.179528 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:23.179457 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 02:31:23.179528 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:23.179522 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f5283dd-ae41-4c1a-a1d3-84285f55e969-networking-console-plugin-cert podName:3f5283dd-ae41-4c1a-a1d3-84285f55e969 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:39.179506555 +0000 UTC m=+87.309748629 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3f5283dd-ae41-4c1a-a1d3-84285f55e969-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-j5465" (UID: "3f5283dd-ae41-4c1a-a1d3-84285f55e969") : secret "networking-console-plugin-cert" not found Sep 09 02:31:23.215903 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:23.215864 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:23.623796 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:23.623765 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:24.240043 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:24.240011 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:24.846901 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:24.846869 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:24.890593 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:24.890565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5a72e3f-0e47-4305-b6f4-b9f294b90913-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:24.890707 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:24.890622 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5a72e3f-0e47-4305-b6f4-b9f294b90913-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:24.890783 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:24.890767 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 02:31:24.890869 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:24.890857 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5a72e3f-0e47-4305-b6f4-b9f294b90913-insights-runtime-extractor-tls podName:b5a72e3f-0e47-4305-b6f4-b9f294b90913 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:40.890837691 +0000 UTC m=+89.021079767 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b5a72e3f-0e47-4305-b6f4-b9f294b90913-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mq79l" (UID: "b5a72e3f-0e47-4305-b6f4-b9f294b90913") : secret "insights-runtime-extractor-tls" not found Sep 09 02:31:24.891215 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:24.891194 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5a72e3f-0e47-4305-b6f4-b9f294b90913-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:25.018202 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:25.018160 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:25.236444 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:25.236402 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:25.617377 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:25.617300 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:25.827147 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:25.827116 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:26.211352 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:26.211322 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:28.202329 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:28.202293 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:28.297145 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:28.297112 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:28.303998 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:28.303983 2577 scope.go:117] "RemoveContainer" containerID="f7f38850b1c6458600ac951b99b13159a1f7765a4df41bad8966999b310eddb6" Sep 09 02:31:28.803015 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:28.802991 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/2.log" Sep 09 02:31:28.803447 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:28.803430 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/1.log" Sep 09 02:31:28.803505 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:28.803461 2577 generic.go:356] "Generic (PLEG): container finished" podID="bbed170a-4fc8-4c57-a0dd-320c37ea878c" containerID="d608c8da21f88bf53025da7ab8fe563de02071d75452695a14973fc730ebcc97" exitCode=255 Sep 09 02:31:28.803554 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:28.803506 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-g849c" event={"ID":"bbed170a-4fc8-4c57-a0dd-320c37ea878c","Type":"ContainerDied","Data":"d608c8da21f88bf53025da7ab8fe563de02071d75452695a14973fc730ebcc97"} Sep 09 02:31:28.803554 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:28.803551 2577 scope.go:117] "RemoveContainer" containerID="f7f38850b1c6458600ac951b99b13159a1f7765a4df41bad8966999b310eddb6" Sep 09 02:31:28.803903 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:28.803872 2577 scope.go:117] "RemoveContainer" containerID="d608c8da21f88bf53025da7ab8fe563de02071d75452695a14973fc730ebcc97" Sep 09 02:31:28.804115 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:28.804043 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-g849c_openshift-console-operator(bbed170a-4fc8-4c57-a0dd-320c37ea878c)\"" pod="openshift-console-operator/console-operator-85b454874b-g849c" podUID="bbed170a-4fc8-4c57-a0dd-320c37ea878c" Sep 09 02:31:28.919830 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:28.919771 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:29.808097 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:29.808072 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/2.log" Sep 09 02:31:32.787895 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:32.787861 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2zpcz" Sep 09 02:31:34.146148 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:34.146027 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-df69fdc44-gjkh5"] Sep 09 02:31:34.308003 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:34.307964 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:34.414180 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:34.414103 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:34.514552 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:34.514522 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:34.550881 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:34.550855 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-g849c" Sep 09 02:31:34.551007 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:34.550905 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-g849c" Sep 09 02:31:34.551217 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:34.551201 2577 scope.go:117] "RemoveContainer" containerID="d608c8da21f88bf53025da7ab8fe563de02071d75452695a14973fc730ebcc97" Sep 09 02:31:34.551390 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:34.551372 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-g849c_openshift-console-operator(bbed170a-4fc8-4c57-a0dd-320c37ea878c)\"" pod="openshift-console-operator/console-operator-85b454874b-g849c" podUID="bbed170a-4fc8-4c57-a0dd-320c37ea878c" Sep 09 02:31:39.097850 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:39.097676 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:39.189509 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:39.189479 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:39.197624 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:39.197600 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3f5283dd-ae41-4c1a-a1d3-84285f55e969-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-j5465\" (UID: \"3f5283dd-ae41-4c1a-a1d3-84285f55e969\") " pod="openshift-network-console/networking-console-plugin-79977f9844-j5465" Sep 09 02:31:39.200043 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:39.200018 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3f5283dd-ae41-4c1a-a1d3-84285f55e969-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-j5465\" (UID: \"3f5283dd-ae41-4c1a-a1d3-84285f55e969\") " pod="openshift-network-console/networking-console-plugin-79977f9844-j5465" Sep 09 02:31:39.265680 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:39.265644 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kjfz6"] Sep 09 02:31:39.288467 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:39.288444 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:39.300531 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:39.300509 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kjfz6"] Sep 09 02:31:39.300652 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:39.300615 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kjfz6" Sep 09 02:31:39.303485 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:39.303466 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-q9xx5\"" Sep 09 02:31:39.303963 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:39.303946 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 09 02:31:39.364516 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:39.364447 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:39.399185 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:39.399153 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f6e44699-f59e-45df-a19a-939054986aeb-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-kjfz6\" (UID: \"f6e44699-f59e-45df-a19a-939054986aeb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kjfz6" Sep 09 02:31:39.431051 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:39.431028 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-rsbbl\"" Sep 09 02:31:39.438557 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:39.438536 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-j5465" Sep 09 02:31:39.500303 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:39.500273 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f6e44699-f59e-45df-a19a-939054986aeb-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-kjfz6\" (UID: \"f6e44699-f59e-45df-a19a-939054986aeb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kjfz6" Sep 09 02:31:39.502638 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:39.502610 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f6e44699-f59e-45df-a19a-939054986aeb-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-kjfz6\" (UID: \"f6e44699-f59e-45df-a19a-939054986aeb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kjfz6" Sep 09 02:31:39.609295 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:39.609255 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kjfz6" Sep 09 02:31:39.736550 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:39.736493 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-j5465"] Sep 09 02:31:39.739386 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:31:39.739361 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5283dd_ae41_4c1a_a1d3_84285f55e969.slice/crio-f8cb1b282fb7aa8d94051a6f36d8c724ace23782c37e02fc2e3efed638319d5b WatchSource:0}: Error finding container f8cb1b282fb7aa8d94051a6f36d8c724ace23782c37e02fc2e3efed638319d5b: Status 404 returned error can't find the container with id f8cb1b282fb7aa8d94051a6f36d8c724ace23782c37e02fc2e3efed638319d5b Sep 09 02:31:39.839705 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:39.839674 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-j5465" event={"ID":"3f5283dd-ae41-4c1a-a1d3-84285f55e969","Type":"ContainerStarted","Data":"f8cb1b282fb7aa8d94051a6f36d8c724ace23782c37e02fc2e3efed638319d5b"} Sep 09 02:31:39.901981 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:39.901913 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kjfz6"] Sep 09 02:31:39.904973 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:31:39.904950 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e44699_f59e_45df_a19a_939054986aeb.slice/crio-ad1bcd56eb6edb3d8944aac77b3c027c014df7089550b5ecdf931174c04765ac WatchSource:0}: Error finding container ad1bcd56eb6edb3d8944aac77b3c027c014df7089550b5ecdf931174c04765ac: Status 404 returned error can't find the container with id ad1bcd56eb6edb3d8944aac77b3c027c014df7089550b5ecdf931174c04765ac Sep 09 02:31:40.849068 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:40.849032 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kjfz6" event={"ID":"f6e44699-f59e-45df-a19a-939054986aeb","Type":"ContainerStarted","Data":"ad1bcd56eb6edb3d8944aac77b3c027c014df7089550b5ecdf931174c04765ac"} Sep 09 02:31:40.910202 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:40.910174 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5a72e3f-0e47-4305-b6f4-b9f294b90913-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:40.912430 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:40.912409 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5a72e3f-0e47-4305-b6f4-b9f294b90913-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mq79l\" (UID: \"b5a72e3f-0e47-4305-b6f4-b9f294b90913\") " pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:41.120112 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:41.120028 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-68fdc\"" Sep 09 02:31:41.127313 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:41.127289 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mq79l" Sep 09 02:31:41.521129 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:41.521099 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mq79l"] Sep 09 02:31:41.523952 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:31:41.523925 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a72e3f_0e47_4305_b6f4_b9f294b90913.slice/crio-7f16864f6d198b3810fd45b5f3e72b83ebaba0ba515700819d34e3dc995203b7 WatchSource:0}: Error finding container 7f16864f6d198b3810fd45b5f3e72b83ebaba0ba515700819d34e3dc995203b7: Status 404 returned error can't find the container with id 7f16864f6d198b3810fd45b5f3e72b83ebaba0ba515700819d34e3dc995203b7 Sep 09 02:31:41.854191 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:41.854100 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kjfz6" event={"ID":"f6e44699-f59e-45df-a19a-939054986aeb","Type":"ContainerStarted","Data":"c8de4e32f3d6d462f0030584e992ac0d9c4fef4e7c2e78dd86e2fb73deb1c8da"} Sep 09 02:31:41.854621 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:41.854276 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kjfz6" Sep 09 02:31:41.855879 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:41.855843 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mq79l" event={"ID":"b5a72e3f-0e47-4305-b6f4-b9f294b90913","Type":"ContainerStarted","Data":"de0887d859ca03ca815ebdc0ab37a2aebe8aeb45d8fe8c44473f0cbe7e5d639a"} Sep 09 02:31:41.856005 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:41.855909 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mq79l" event={"ID":"b5a72e3f-0e47-4305-b6f4-b9f294b90913","Type":"ContainerStarted","Data":"7f16864f6d198b3810fd45b5f3e72b83ebaba0ba515700819d34e3dc995203b7"} Sep 09 02:31:41.857485 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:41.857463 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-j5465" event={"ID":"3f5283dd-ae41-4c1a-a1d3-84285f55e969","Type":"ContainerStarted","Data":"e6d685f507beb18106e63bc8e84781f1875b54d679e5219602e7a6fbe5508b40"} Sep 09 02:31:41.859237 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:41.859219 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kjfz6" Sep 09 02:31:41.871913 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:41.871869 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kjfz6" podStartSLOduration=1.505389418 podStartE2EDuration="2.871854472s" totalImagesPullingTime="1.366465054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:41.871784484 +0000 UTC m=+90.002026580" watchObservedRunningTime="2026-09-09 02:31:41.871854472 +0000 UTC m=+90.002096568" Sep 09 02:31:41.893420 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:41.893376 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-j5465" podStartSLOduration=33.74981085 podStartE2EDuration="34.893361186s" totalImagesPullingTime="1.143550336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:41.89321156 +0000 UTC m=+90.023453656" watchObservedRunningTime="2026-09-09 02:31:41.893361186 +0000 UTC m=+90.023603282" Sep 09 02:31:42.010949 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:42.010917 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:42.080407 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:42.080378 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:42.295069 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:42.295042 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:42.396508 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:42.396479 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:42.864413 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:42.864336 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mq79l" event={"ID":"b5a72e3f-0e47-4305-b6f4-b9f294b90913","Type":"ContainerStarted","Data":"de0d7b2a34c785b21e6d04cdc6ebee01aab7459575dae3dc156c57c355a8badb"} Sep 09 02:31:42.935322 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:42.935275 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:43.215261 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:43.215222 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:43.286417 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:43.286372 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:43.869277 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:43.869185 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mq79l" event={"ID":"b5a72e3f-0e47-4305-b6f4-b9f294b90913","Type":"ContainerStarted","Data":"90253009ee3b16125b179d0af39680e00a5f2bdf9d639efbff8c0902eeab2ca0"} Sep 09 02:31:43.891198 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:43.891143 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mq79l" podStartSLOduration=34.033110723 podStartE2EDuration="35.891129057s" totalImagesPullingTime="1.858018334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:31:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:43.890259848 +0000 UTC m=+92.020501944" watchObservedRunningTime="2026-09-09 02:31:43.891129057 +0000 UTC m=+92.021371153" Sep 09 02:31:44.152055 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:44.152029 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:45.534828 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:45.534794 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:45.723346 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:45.723313 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:45.836635 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:45.836561 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:46.633234 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:46.633203 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:46.733527 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:46.733496 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:47.349910 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.349881 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:47.475719 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.475691 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:47.596221 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.596191 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:47.714104 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.714074 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:47.755297 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.755269 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bbcsl"] Sep 09 02:31:47.757492 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.757477 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.759374 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.759308 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j"] Sep 09 02:31:47.760680 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.760656 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tj6kt\"" Sep 09 02:31:47.760784 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.760692 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 09 02:31:47.761006 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.760992 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 09 02:31:47.761006 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.760999 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 09 02:31:47.761111 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.761013 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 09 02:31:47.761791 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.761775 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 09 02:31:47.762526 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.762510 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-9h959"] Sep 09 02:31:47.762629 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.762616 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" Sep 09 02:31:47.763118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.763100 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 09 02:31:47.764610 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.764593 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9h959" Sep 09 02:31:47.765526 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.765405 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 09 02:31:47.766492 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.766466 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8g2bl\"" Sep 09 02:31:47.766914 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.766894 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 09 02:31:47.766988 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.766923 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 09 02:31:47.770497 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.770478 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 09 02:31:47.770586 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.770535 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 09 02:31:47.772134 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.772120 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-d77bx\"" Sep 09 02:31:47.778776 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.778756 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-9h959"] Sep 09 02:31:47.779982 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.779961 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j"] Sep 09 02:31:47.851421 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.851391 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:47.861510 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.861483 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/833e16d1-4060-4d33-aff7-945b2a7c2786-node-exporter-textfile\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.861622 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.861544 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fb28dabe-ca30-4d6c-83bb-271e6751e5e1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-ztr5j\" (UID: \"fb28dabe-ca30-4d6c-83bb-271e6751e5e1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" Sep 09 02:31:47.861622 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.861588 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpccm\" (UniqueName: \"kubernetes.io/projected/654087bd-60b8-4b72-8104-ce90acbb0379-kube-api-access-lpccm\") pod \"openshift-state-metrics-657fd6c54-9h959\" (UID: \"654087bd-60b8-4b72-8104-ce90acbb0379\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9h959" Sep 09 02:31:47.861622 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.861613 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/833e16d1-4060-4d33-aff7-945b2a7c2786-sys\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.861772 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.861668 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/654087bd-60b8-4b72-8104-ce90acbb0379-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-9h959\" (UID: \"654087bd-60b8-4b72-8104-ce90acbb0379\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9h959" Sep 09 02:31:47.861772 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.861696 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/833e16d1-4060-4d33-aff7-945b2a7c2786-node-exporter-accelerators-collector-config\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.861772 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.861724 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb28dabe-ca30-4d6c-83bb-271e6751e5e1-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-ztr5j\" (UID: \"fb28dabe-ca30-4d6c-83bb-271e6751e5e1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" Sep 09 02:31:47.861893 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.861768 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb28dabe-ca30-4d6c-83bb-271e6751e5e1-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-ztr5j\" (UID: \"fb28dabe-ca30-4d6c-83bb-271e6751e5e1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" Sep 09 02:31:47.861893 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.861802 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/833e16d1-4060-4d33-aff7-945b2a7c2786-root\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.861893 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.861846 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxhpm\" (UniqueName: \"kubernetes.io/projected/fb28dabe-ca30-4d6c-83bb-271e6751e5e1-kube-api-access-fxhpm\") pod \"kube-state-metrics-74cb57df7-ztr5j\" (UID: \"fb28dabe-ca30-4d6c-83bb-271e6751e5e1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" Sep 09 02:31:47.861893 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.861875 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/654087bd-60b8-4b72-8104-ce90acbb0379-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-9h959\" (UID: \"654087bd-60b8-4b72-8104-ce90acbb0379\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9h959" Sep 09 02:31:47.862067 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.861930 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/833e16d1-4060-4d33-aff7-945b2a7c2786-metrics-client-ca\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.862067 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.861968 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fb28dabe-ca30-4d6c-83bb-271e6751e5e1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-ztr5j\" (UID: \"fb28dabe-ca30-4d6c-83bb-271e6751e5e1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" Sep 09 02:31:47.862067 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.862005 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/833e16d1-4060-4d33-aff7-945b2a7c2786-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.862067 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.862049 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/833e16d1-4060-4d33-aff7-945b2a7c2786-node-exporter-tls\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.862197 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.862071 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/833e16d1-4060-4d33-aff7-945b2a7c2786-node-exporter-wtmp\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.862197 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.862092 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fb28dabe-ca30-4d6c-83bb-271e6751e5e1-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-ztr5j\" (UID: \"fb28dabe-ca30-4d6c-83bb-271e6751e5e1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" Sep 09 02:31:47.862197 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.862112 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/654087bd-60b8-4b72-8104-ce90acbb0379-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-9h959\" (UID: \"654087bd-60b8-4b72-8104-ce90acbb0379\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9h959" Sep 09 02:31:47.862197 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.862127 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5dwf\" (UniqueName: \"kubernetes.io/projected/833e16d1-4060-4d33-aff7-945b2a7c2786-kube-api-access-r5dwf\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.962943 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.962907 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/654087bd-60b8-4b72-8104-ce90acbb0379-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-9h959\" (UID: \"654087bd-60b8-4b72-8104-ce90acbb0379\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9h959" Sep 09 02:31:47.963117 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.962950 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/833e16d1-4060-4d33-aff7-945b2a7c2786-node-exporter-accelerators-collector-config\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.963117 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.963078 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb28dabe-ca30-4d6c-83bb-271e6751e5e1-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-ztr5j\" (UID: \"fb28dabe-ca30-4d6c-83bb-271e6751e5e1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" Sep 09 02:31:47.963117 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.963112 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb28dabe-ca30-4d6c-83bb-271e6751e5e1-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-ztr5j\" (UID: \"fb28dabe-ca30-4d6c-83bb-271e6751e5e1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" Sep 09 02:31:47.963304 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.963139 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/833e16d1-4060-4d33-aff7-945b2a7c2786-root\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.963304 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.963165 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxhpm\" (UniqueName: \"kubernetes.io/projected/fb28dabe-ca30-4d6c-83bb-271e6751e5e1-kube-api-access-fxhpm\") pod \"kube-state-metrics-74cb57df7-ztr5j\" (UID: \"fb28dabe-ca30-4d6c-83bb-271e6751e5e1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" Sep 09 02:31:47.963304 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.963191 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/654087bd-60b8-4b72-8104-ce90acbb0379-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-9h959\" (UID: \"654087bd-60b8-4b72-8104-ce90acbb0379\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9h959" Sep 09 02:31:47.963304 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.963231 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/833e16d1-4060-4d33-aff7-945b2a7c2786-metrics-client-ca\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.963304 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.963228 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/833e16d1-4060-4d33-aff7-945b2a7c2786-root\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.963304 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.963262 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fb28dabe-ca30-4d6c-83bb-271e6751e5e1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-ztr5j\" (UID: \"fb28dabe-ca30-4d6c-83bb-271e6751e5e1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" Sep 09 02:31:47.963304 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:47.963237 2577 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 09 02:31:47.963655 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:47.963364 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb28dabe-ca30-4d6c-83bb-271e6751e5e1-kube-state-metrics-tls podName:fb28dabe-ca30-4d6c-83bb-271e6751e5e1 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:48.463343377 +0000 UTC m=+96.593585464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/fb28dabe-ca30-4d6c-83bb-271e6751e5e1-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-ztr5j" (UID: "fb28dabe-ca30-4d6c-83bb-271e6751e5e1") : secret "kube-state-metrics-tls" not found Sep 09 02:31:47.963655 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.963628 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/833e16d1-4060-4d33-aff7-945b2a7c2786-node-exporter-accelerators-collector-config\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.963757 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.963657 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/833e16d1-4060-4d33-aff7-945b2a7c2786-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.963757 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.963715 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/833e16d1-4060-4d33-aff7-945b2a7c2786-node-exporter-tls\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.963757 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.963742 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/833e16d1-4060-4d33-aff7-945b2a7c2786-node-exporter-wtmp\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.963937 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.963778 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fb28dabe-ca30-4d6c-83bb-271e6751e5e1-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-ztr5j\" (UID: \"fb28dabe-ca30-4d6c-83bb-271e6751e5e1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" Sep 09 02:31:47.963937 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.963812 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/654087bd-60b8-4b72-8104-ce90acbb0379-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-9h959\" (UID: \"654087bd-60b8-4b72-8104-ce90acbb0379\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9h959" Sep 09 02:31:47.963937 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.963856 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5dwf\" (UniqueName: \"kubernetes.io/projected/833e16d1-4060-4d33-aff7-945b2a7c2786-kube-api-access-r5dwf\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.963937 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.963882 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/833e16d1-4060-4d33-aff7-945b2a7c2786-node-exporter-textfile\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.963937 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.963910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fb28dabe-ca30-4d6c-83bb-271e6751e5e1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-ztr5j\" (UID: \"fb28dabe-ca30-4d6c-83bb-271e6751e5e1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" Sep 09 02:31:47.963937 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.963924 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb28dabe-ca30-4d6c-83bb-271e6751e5e1-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-ztr5j\" (UID: \"fb28dabe-ca30-4d6c-83bb-271e6751e5e1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" Sep 09 02:31:47.964260 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.963944 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpccm\" (UniqueName: \"kubernetes.io/projected/654087bd-60b8-4b72-8104-ce90acbb0379-kube-api-access-lpccm\") pod \"openshift-state-metrics-657fd6c54-9h959\" (UID: \"654087bd-60b8-4b72-8104-ce90acbb0379\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9h959" Sep 09 02:31:47.964260 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.963966 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/833e16d1-4060-4d33-aff7-945b2a7c2786-sys\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.964260 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.963974 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fb28dabe-ca30-4d6c-83bb-271e6751e5e1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-ztr5j\" (UID: \"fb28dabe-ca30-4d6c-83bb-271e6751e5e1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" Sep 09 02:31:47.964260 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.963977 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/833e16d1-4060-4d33-aff7-945b2a7c2786-metrics-client-ca\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.964260 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.964064 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/833e16d1-4060-4d33-aff7-945b2a7c2786-sys\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.964260 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.964135 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/654087bd-60b8-4b72-8104-ce90acbb0379-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-9h959\" (UID: \"654087bd-60b8-4b72-8104-ce90acbb0379\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9h959" Sep 09 02:31:47.964260 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.964178 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/833e16d1-4060-4d33-aff7-945b2a7c2786-node-exporter-wtmp\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.964260 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:47.964224 2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 09 02:31:47.964260 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.964243 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/833e16d1-4060-4d33-aff7-945b2a7c2786-node-exporter-textfile\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.964709 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.964267 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fb28dabe-ca30-4d6c-83bb-271e6751e5e1-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-ztr5j\" (UID: \"fb28dabe-ca30-4d6c-83bb-271e6751e5e1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" Sep 09 02:31:47.964709 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:47.964320 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/833e16d1-4060-4d33-aff7-945b2a7c2786-node-exporter-tls podName:833e16d1-4060-4d33-aff7-945b2a7c2786 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:48.46430239 +0000 UTC m=+96.594544477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/833e16d1-4060-4d33-aff7-945b2a7c2786-node-exporter-tls") pod "node-exporter-bbcsl" (UID: "833e16d1-4060-4d33-aff7-945b2a7c2786") : secret "node-exporter-tls" not found Sep 09 02:31:47.965632 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.965609 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/654087bd-60b8-4b72-8104-ce90acbb0379-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-9h959\" (UID: \"654087bd-60b8-4b72-8104-ce90acbb0379\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9h959" Sep 09 02:31:47.966262 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.966245 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/833e16d1-4060-4d33-aff7-945b2a7c2786-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.966262 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.966253 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/654087bd-60b8-4b72-8104-ce90acbb0379-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-9h959\" (UID: \"654087bd-60b8-4b72-8104-ce90acbb0379\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9h959" Sep 09 02:31:47.966448 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.966429 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fb28dabe-ca30-4d6c-83bb-271e6751e5e1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-ztr5j\" (UID: \"fb28dabe-ca30-4d6c-83bb-271e6751e5e1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" Sep 09 02:31:47.976135 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.976109 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpccm\" (UniqueName: \"kubernetes.io/projected/654087bd-60b8-4b72-8104-ce90acbb0379-kube-api-access-lpccm\") pod \"openshift-state-metrics-657fd6c54-9h959\" (UID: \"654087bd-60b8-4b72-8104-ce90acbb0379\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9h959" Sep 09 02:31:47.976420 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.976399 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5dwf\" (UniqueName: \"kubernetes.io/projected/833e16d1-4060-4d33-aff7-945b2a7c2786-kube-api-access-r5dwf\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:47.976520 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.976506 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxhpm\" (UniqueName: \"kubernetes.io/projected/fb28dabe-ca30-4d6c-83bb-271e6751e5e1-kube-api-access-fxhpm\") pod \"kube-state-metrics-74cb57df7-ztr5j\" (UID: \"fb28dabe-ca30-4d6c-83bb-271e6751e5e1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" Sep 09 02:31:47.991031 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:47.991009 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:48.084763 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.084735 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9h959" Sep 09 02:31:48.095315 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.095286 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:48.196939 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.196906 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:48.304177 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.304097 2577 scope.go:117] "RemoveContainer" containerID="d608c8da21f88bf53025da7ab8fe563de02071d75452695a14973fc730ebcc97" Sep 09 02:31:48.304319 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:48.304251 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-g849c_openshift-console-operator(bbed170a-4fc8-4c57-a0dd-320c37ea878c)\"" pod="openshift-console-operator/console-operator-85b454874b-g849c" podUID="bbed170a-4fc8-4c57-a0dd-320c37ea878c" Sep 09 02:31:48.360097 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.360067 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-9h959"] Sep 09 02:31:48.362774 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:31:48.362740 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654087bd_60b8_4b72_8104_ce90acbb0379.slice/crio-0c6a3b46b167456e8b385b6e839dc718d238d1c005a245dfa7ff8d771ae6693f WatchSource:0}: Error finding container 0c6a3b46b167456e8b385b6e839dc718d238d1c005a245dfa7ff8d771ae6693f: Status 404 returned error can't find the container with id 0c6a3b46b167456e8b385b6e839dc718d238d1c005a245dfa7ff8d771ae6693f Sep 09 02:31:48.373013 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.372989 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:48.447591 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.447561 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:48.467444 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.467424 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/833e16d1-4060-4d33-aff7-945b2a7c2786-node-exporter-tls\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:48.467535 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.467483 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb28dabe-ca30-4d6c-83bb-271e6751e5e1-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-ztr5j\" (UID: \"fb28dabe-ca30-4d6c-83bb-271e6751e5e1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" Sep 09 02:31:48.469537 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.469517 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb28dabe-ca30-4d6c-83bb-271e6751e5e1-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-ztr5j\" (UID: \"fb28dabe-ca30-4d6c-83bb-271e6751e5e1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" Sep 09 02:31:48.469855 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.469838 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/833e16d1-4060-4d33-aff7-945b2a7c2786-node-exporter-tls\") pod \"node-exporter-bbcsl\" (UID: \"833e16d1-4060-4d33-aff7-945b2a7c2786\") " pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:48.672245 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.672214 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bbcsl" Sep 09 02:31:48.678281 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.678252 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" Sep 09 02:31:48.681560 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:31:48.681536 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833e16d1_4060_4d33_aff7_945b2a7c2786.slice/crio-db28c6ee2d5768d8ad71a7fb3dafde4e35d777ea54c8c967f183c6b70d0b87c1 WatchSource:0}: Error finding container db28c6ee2d5768d8ad71a7fb3dafde4e35d777ea54c8c967f183c6b70d0b87c1: Status 404 returned error can't find the container with id db28c6ee2d5768d8ad71a7fb3dafde4e35d777ea54c8c967f183c6b70d0b87c1 Sep 09 02:31:48.736043 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.735724 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:48.763086 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.763060 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zr2ql" Sep 09 02:31:48.839873 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.839844 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:48.861926 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.861895 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 02:31:48.865229 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.865210 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:48.867712 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.867691 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 09 02:31:48.868739 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.868722 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 09 02:31:48.868843 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.868726 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 09 02:31:48.868843 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.868757 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 09 02:31:48.869046 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.869021 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 09 02:31:48.869111 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.869056 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 09 02:31:48.869111 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.869089 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 09 02:31:48.869215 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.869181 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 09 02:31:48.869417 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.869401 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ssw27\"" Sep 09 02:31:48.869459 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.869451 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 09 02:31:48.887632 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.887599 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 02:31:48.888411 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.888380 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bbcsl" event={"ID":"833e16d1-4060-4d33-aff7-945b2a7c2786","Type":"ContainerStarted","Data":"db28c6ee2d5768d8ad71a7fb3dafde4e35d777ea54c8c967f183c6b70d0b87c1"} Sep 09 02:31:48.890482 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.890460 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9h959" event={"ID":"654087bd-60b8-4b72-8104-ce90acbb0379","Type":"ContainerStarted","Data":"4f16b7c926d39a8758f3e70561d14449fd8a330266edf99ac7d4d5989b5958e3"} Sep 09 02:31:48.890573 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.890506 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9h959" event={"ID":"654087bd-60b8-4b72-8104-ce90acbb0379","Type":"ContainerStarted","Data":"d74bcd51564538833d994bea40c447eeae76a55c65cf97b7e2c24b165473591f"} Sep 09 02:31:48.890573 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.890527 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9h959" event={"ID":"654087bd-60b8-4b72-8104-ce90acbb0379","Type":"ContainerStarted","Data":"0c6a3b46b167456e8b385b6e839dc718d238d1c005a245dfa7ff8d771ae6693f"} Sep 09 02:31:48.972286 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.972128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxlxf\" (UniqueName: \"kubernetes.io/projected/4890a884-dbfb-4177-b14b-2a886198ac47-kube-api-access-cxlxf\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:48.972286 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.972188 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-config-volume\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:48.972501 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.972289 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:48.972501 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.972327 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4890a884-dbfb-4177-b14b-2a886198ac47-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:48.972501 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.972365 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:48.972501 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.972399 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4890a884-dbfb-4177-b14b-2a886198ac47-config-out\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:48.972501 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.972429 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:48.972727 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.972476 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:48.972727 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.972557 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4890a884-dbfb-4177-b14b-2a886198ac47-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:48.972727 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.972588 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-web-config\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:48.972727 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.972675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4890a884-dbfb-4177-b14b-2a886198ac47-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:48.972727 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.972710 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:48.972970 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.972774 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4890a884-dbfb-4177-b14b-2a886198ac47-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:48.974044 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:48.974013 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j"] Sep 09 02:31:48.981106 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:31:48.981076 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb28dabe_ca30_4d6c_83bb_271e6751e5e1.slice/crio-2e891627fb5f6feb3fe8937b02a6b3d58e7e52d9190cef23be0e5e96fd873e44 WatchSource:0}: Error finding container 2e891627fb5f6feb3fe8937b02a6b3d58e7e52d9190cef23be0e5e96fd873e44: Status 404 returned error can't find the container with id 2e891627fb5f6feb3fe8937b02a6b3d58e7e52d9190cef23be0e5e96fd873e44 Sep 09 02:31:49.055429 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.055392 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:49.073416 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.073375 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4890a884-dbfb-4177-b14b-2a886198ac47-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.073416 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.073414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.073645 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.073442 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4890a884-dbfb-4177-b14b-2a886198ac47-config-out\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.073645 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.073465 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.073645 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.073500 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.073645 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.073531 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4890a884-dbfb-4177-b14b-2a886198ac47-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.073645 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.073554 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-web-config\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.073645 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.073576 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4890a884-dbfb-4177-b14b-2a886198ac47-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.073645 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.073597 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.073645 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.073630 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4890a884-dbfb-4177-b14b-2a886198ac47-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.074051 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.073658 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxlxf\" (UniqueName: \"kubernetes.io/projected/4890a884-dbfb-4177-b14b-2a886198ac47-kube-api-access-cxlxf\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.074051 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.073685 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-config-volume\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.074051 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.073808 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.074878 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.074287 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4890a884-dbfb-4177-b14b-2a886198ac47-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.074878 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.074305 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4890a884-dbfb-4177-b14b-2a886198ac47-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.075050 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.075019 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4890a884-dbfb-4177-b14b-2a886198ac47-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.076846 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.076771 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.077213 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.077185 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4890a884-dbfb-4177-b14b-2a886198ac47-config-out\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.077377 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.077353 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.077841 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.077785 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.078219 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.078194 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.078684 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.078659 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.079101 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.079080 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-config-volume\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.079308 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.079149 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4890a884-dbfb-4177-b14b-2a886198ac47-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.079662 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.079643 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-web-config\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.083628 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.083608 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxlxf\" (UniqueName: \"kubernetes.io/projected/4890a884-dbfb-4177-b14b-2a886198ac47-kube-api-access-cxlxf\") pod \"alertmanager-main-0\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.174886 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.174840 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:31:49.467259 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.467224 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 02:31:49.471973 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:31:49.471942 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4890a884_dbfb_4177_b14b_2a886198ac47.slice/crio-b700819d757c93a868acc61dcb94e4d0cbf6e83efa4c1ab5d38898a9cdd1e2f6 WatchSource:0}: Error finding container b700819d757c93a868acc61dcb94e4d0cbf6e83efa4c1ab5d38898a9cdd1e2f6: Status 404 returned error can't find the container with id b700819d757c93a868acc61dcb94e4d0cbf6e83efa4c1ab5d38898a9cdd1e2f6 Sep 09 02:31:49.891039 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.891005 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf"] Sep 09 02:31:49.894653 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.894618 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:49.898619 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.898179 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 09 02:31:49.898619 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.898378 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 09 02:31:49.898619 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.898447 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 09 02:31:49.898619 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.898574 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-12o0gbplihp3l\"" Sep 09 02:31:49.898619 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.898592 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 09 02:31:49.899075 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.898675 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-bsz8b\"" Sep 09 02:31:49.899075 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.898916 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 09 02:31:49.899075 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.898926 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bbcsl" event={"ID":"833e16d1-4060-4d33-aff7-945b2a7c2786","Type":"ContainerStarted","Data":"803cf443aeeb24e0211d82bd81eeb1e05513e8aa52e11629a0a3dda191023ffb"} Sep 09 02:31:49.902620 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.902444 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9h959" event={"ID":"654087bd-60b8-4b72-8104-ce90acbb0379","Type":"ContainerStarted","Data":"90c7e6b9b9336920a90495376fad61cec7d3fa98fa7c8d77f43cdd2a4f69382c"} Sep 09 02:31:49.905053 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.904957 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4890a884-dbfb-4177-b14b-2a886198ac47","Type":"ContainerStarted","Data":"b700819d757c93a868acc61dcb94e4d0cbf6e83efa4c1ab5d38898a9cdd1e2f6"} Sep 09 02:31:49.906566 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.906527 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" event={"ID":"fb28dabe-ca30-4d6c-83bb-271e6751e5e1","Type":"ContainerStarted","Data":"2e891627fb5f6feb3fe8937b02a6b3d58e7e52d9190cef23be0e5e96fd873e44"} Sep 09 02:31:49.909341 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.909319 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf"] Sep 09 02:31:49.980165 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.980103 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-9h959" podStartSLOduration=1.96645867 podStartE2EDuration="2.9800797s" totalImagesPullingTime="1.01362103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:49.978523368 +0000 UTC m=+98.108765464" watchObservedRunningTime="2026-09-09 02:31:49.9800797 +0000 UTC m=+98.110321798" Sep 09 02:31:49.982960 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.982929 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:49.983102 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.982977 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:49.983158 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.983134 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-secret-thanos-querier-tls\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:49.983224 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.983176 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:49.983224 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.983207 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l8sc\" (UniqueName: \"kubernetes.io/projected/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-kube-api-access-9l8sc\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:49.983462 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.983426 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-secret-grpc-tls\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:49.983568 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.983488 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:49.983568 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:49.983518 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-metrics-client-ca\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:50.084990 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.084910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-secret-thanos-querier-tls\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:50.084990 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.084947 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:50.084990 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.084969 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l8sc\" (UniqueName: \"kubernetes.io/projected/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-kube-api-access-9l8sc\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:50.085261 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.084994 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-secret-grpc-tls\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:50.085261 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.085025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:50.085261 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.085052 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-metrics-client-ca\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:50.085261 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.085098 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:50.085261 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.085126 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:50.086151 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.086076 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-metrics-client-ca\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:50.087939 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.087912 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:50.088049 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.087969 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-secret-thanos-querier-tls\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:50.088109 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.088047 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:50.088212 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.088192 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-secret-grpc-tls\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:50.088357 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.088311 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:50.088751 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.088727 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:50.094298 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.094267 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l8sc\" (UniqueName: \"kubernetes.io/projected/15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0-kube-api-access-9l8sc\") pod \"thanos-querier-5cb9bb7cb4-mfqnf\" (UID: \"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0\") " pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:50.235602 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.235562 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:50.436633 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.436605 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:50.536913 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.536880 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:50.640843 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.640796 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:50.746492 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.746421 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:50.845838 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.845758 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:50.921550 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.921515 2577 generic.go:356] "Generic (PLEG): container finished" podID="833e16d1-4060-4d33-aff7-945b2a7c2786" containerID="803cf443aeeb24e0211d82bd81eeb1e05513e8aa52e11629a0a3dda191023ffb" exitCode=0 Sep 09 02:31:50.922037 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.921593 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bbcsl" event={"ID":"833e16d1-4060-4d33-aff7-945b2a7c2786","Type":"ContainerDied","Data":"803cf443aeeb24e0211d82bd81eeb1e05513e8aa52e11629a0a3dda191023ffb"} Sep 09 02:31:50.924010 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.923981 2577 generic.go:356] "Generic (PLEG): container finished" podID="4890a884-dbfb-4177-b14b-2a886198ac47" containerID="40bd3e30bafeddee487114980193be1f082d702baf11c70e2e33570e4a9ba3d8" exitCode=0 Sep 09 02:31:50.924164 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.924064 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4890a884-dbfb-4177-b14b-2a886198ac47","Type":"ContainerDied","Data":"40bd3e30bafeddee487114980193be1f082d702baf11c70e2e33570e4a9ba3d8"} Sep 09 02:31:50.927653 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:50.927519 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" event={"ID":"fb28dabe-ca30-4d6c-83bb-271e6751e5e1","Type":"ContainerStarted","Data":"fb9e7b55d54f63fc2c2a4c7581d7e0cf03516f5ba23c6b234f5376be651f3e94"} Sep 09 02:31:51.060849 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:51.060802 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf"] Sep 09 02:31:51.065434 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:31:51.065408 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15592ffc_8bd5_4c4d_a69b_6f2d24e8e6d0.slice/crio-b30b8fd6dc059466fc35e08c907ea50da09fb498828e2421f5010834e208ad1a WatchSource:0}: Error finding container b30b8fd6dc059466fc35e08c907ea50da09fb498828e2421f5010834e208ad1a: Status 404 returned error can't find the container with id b30b8fd6dc059466fc35e08c907ea50da09fb498828e2421f5010834e208ad1a Sep 09 02:31:51.159162 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:51.159126 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:51.788946 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:51.788855 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:51.936051 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:51.936012 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bbcsl" event={"ID":"833e16d1-4060-4d33-aff7-945b2a7c2786","Type":"ContainerStarted","Data":"8ac331939240c6626873b7780f2ae1bfedd41e10c2679bd228b0c64520c40abf"} Sep 09 02:31:51.936522 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:51.936073 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bbcsl" event={"ID":"833e16d1-4060-4d33-aff7-945b2a7c2786","Type":"ContainerStarted","Data":"ba170a2954a8271f4b6d70f97d9e12f09d11b643088c94b3132c87ee69105497"} Sep 09 02:31:51.937804 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:51.937776 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" event={"ID":"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0","Type":"ContainerStarted","Data":"b30b8fd6dc059466fc35e08c907ea50da09fb498828e2421f5010834e208ad1a"} Sep 09 02:31:51.940396 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:51.940371 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" event={"ID":"fb28dabe-ca30-4d6c-83bb-271e6751e5e1","Type":"ContainerStarted","Data":"406bfddccdfc63f55ea1eb3cf1dffcca2c65cc5b5797ca13eab29338d7c34b0a"} Sep 09 02:31:51.940494 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:51.940427 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" event={"ID":"fb28dabe-ca30-4d6c-83bb-271e6751e5e1","Type":"ContainerStarted","Data":"1fd8e1d53c904d45c8794ca4d000b82f4c38c85e6f026dc31f47179aeeb484ed"} Sep 09 02:31:51.962471 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:51.962407 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bbcsl" podStartSLOduration=3.81741053 podStartE2EDuration="4.962385432s" totalImagesPullingTime="1.070663492s" totalInitContainerRuntime="74.31141ms" isStatefulPod=true podCreationTimestamp="2026-09-09 02:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:51.961959211 +0000 UTC m=+100.092201309" watchObservedRunningTime="2026-09-09 02:31:51.962385432 +0000 UTC m=+100.092627551" Sep 09 02:31:51.986942 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:51.986888 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-ztr5j" podStartSLOduration=3.21216366 podStartE2EDuration="4.986866904s" totalImagesPullingTime="1.774703244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:51.98644858 +0000 UTC m=+100.116690676" watchObservedRunningTime="2026-09-09 02:31:51.986866904 +0000 UTC m=+100.117109001" Sep 09 02:31:52.511115 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:52.511082 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:52.515413 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:52.515391 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-g26j9"] Sep 09 02:31:52.529571 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:52.529546 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-g26j9" Sep 09 02:31:52.532258 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:52.532233 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-g26j9"] Sep 09 02:31:52.532694 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:52.532672 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-pz8q6\"" Sep 09 02:31:52.532851 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:52.532767 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 09 02:31:52.608387 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:52.608357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b01cd589-629e-4438-b7c0-758160094ef9-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-g26j9\" (UID: \"b01cd589-629e-4438-b7c0-758160094ef9\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-g26j9" Sep 09 02:31:52.608523 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:52.608402 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p4nc\" (UniqueName: \"kubernetes.io/projected/b01cd589-629e-4438-b7c0-758160094ef9-kube-api-access-6p4nc\") pod \"monitoring-plugin-85bbbbbf4-g26j9\" (UID: \"b01cd589-629e-4438-b7c0-758160094ef9\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-g26j9" Sep 09 02:31:52.709730 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:52.709691 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b01cd589-629e-4438-b7c0-758160094ef9-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-g26j9\" (UID: \"b01cd589-629e-4438-b7c0-758160094ef9\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-g26j9" Sep 09 02:31:52.709918 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:52.709763 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p4nc\" (UniqueName: \"kubernetes.io/projected/b01cd589-629e-4438-b7c0-758160094ef9-kube-api-access-6p4nc\") pod \"monitoring-plugin-85bbbbbf4-g26j9\" (UID: \"b01cd589-629e-4438-b7c0-758160094ef9\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-g26j9" Sep 09 02:31:52.709918 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:52.709867 2577 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 09 02:31:52.710039 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:31:52.709998 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b01cd589-629e-4438-b7c0-758160094ef9-monitoring-plugin-cert podName:b01cd589-629e-4438-b7c0-758160094ef9 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:53.209978201 +0000 UTC m=+101.340220274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/b01cd589-629e-4438-b7c0-758160094ef9-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-g26j9" (UID: "b01cd589-629e-4438-b7c0-758160094ef9") : secret "monitoring-plugin-cert" not found Sep 09 02:31:52.719371 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:52.719339 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p4nc\" (UniqueName: \"kubernetes.io/projected/b01cd589-629e-4438-b7c0-758160094ef9-kube-api-access-6p4nc\") pod \"monitoring-plugin-85bbbbbf4-g26j9\" (UID: \"b01cd589-629e-4438-b7c0-758160094ef9\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-g26j9" Sep 09 02:31:52.933280 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:52.933245 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:53.215285 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:53.215198 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b01cd589-629e-4438-b7c0-758160094ef9-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-g26j9\" (UID: \"b01cd589-629e-4438-b7c0-758160094ef9\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-g26j9" Sep 09 02:31:53.217583 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:53.217561 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b01cd589-629e-4438-b7c0-758160094ef9-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-g26j9\" (UID: \"b01cd589-629e-4438-b7c0-758160094ef9\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-g26j9" Sep 09 02:31:53.442029 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:53.441996 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-g26j9" Sep 09 02:31:53.857212 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:53.857111 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-g26j9"] Sep 09 02:31:53.955534 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:53.955495 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" event={"ID":"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0","Type":"ContainerStarted","Data":"f424646af8ab0082d7b23bf2f7b8a247c71af0faf07dd8e9b515a6927c157105"} Sep 09 02:31:53.955703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:53.955586 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" event={"ID":"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0","Type":"ContainerStarted","Data":"ed04c0777c14ade85b9cff56215f62a285d061b5a74b93bd7bef74e1c766e817"} Sep 09 02:31:53.955703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:53.955656 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" event={"ID":"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0","Type":"ContainerStarted","Data":"6af4c64482d5beda7d7ea4aa4dcb0923313c8f0df1aee28eec67fb5ab38c96d9"} Sep 09 02:31:53.959775 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:53.959746 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4890a884-dbfb-4177-b14b-2a886198ac47","Type":"ContainerStarted","Data":"5c364595689175bab0e9d4af3bd7764ec469a0265f152167d09293428de9c432"} Sep 09 02:31:53.959883 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:53.959840 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4890a884-dbfb-4177-b14b-2a886198ac47","Type":"ContainerStarted","Data":"20c543eb80bdac022269b3fdc08bfd17ab8afa0b22d71d26e328a9316e638afd"} Sep 09 02:31:53.959957 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:53.959885 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4890a884-dbfb-4177-b14b-2a886198ac47","Type":"ContainerStarted","Data":"075feaecc46198cedab3ffe885a171402398eca77cc1efedbe6e7657f321bef9"} Sep 09 02:31:53.961655 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:53.961632 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-g26j9" event={"ID":"b01cd589-629e-4438-b7c0-758160094ef9","Type":"ContainerStarted","Data":"e1f64fece393b60ed91a6c5f773d45638798012e3dbcd43e7a3f2dfbedf40d85"} Sep 09 02:31:54.073269 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.073242 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 02:31:54.077051 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.076085 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.078925 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.078902 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:54.081090 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.081056 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 09 02:31:54.081314 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.081059 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 09 02:31:54.081314 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.081215 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 09 02:31:54.081466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.081318 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tbbb2\"" Sep 09 02:31:54.081594 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.081578 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 09 02:31:54.081663 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.081590 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 09 02:31:54.082334 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.082319 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 09 02:31:54.082415 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.082402 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 09 02:31:54.082487 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.082421 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 09 02:31:54.082487 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.082424 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 09 02:31:54.083083 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.083065 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1bvfn8k5em701\"" Sep 09 02:31:54.083543 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.083525 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 09 02:31:54.083648 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.083562 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 09 02:31:54.083768 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.083753 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 09 02:31:54.083908 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.083757 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 09 02:31:54.085698 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.085665 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 09 02:31:54.087961 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.087946 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 09 02:31:54.102118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.102066 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 02:31:54.224474 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.224441 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.224474 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.224481 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-config\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.224906 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.224507 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.224906 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.224527 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/506453e5-4dc2-4acb-a624-4d0cad7bd388-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.224906 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.224542 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.224906 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.224606 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-web-config\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.224906 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.224621 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.224906 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.224637 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtmv6\" (UniqueName: \"kubernetes.io/projected/506453e5-4dc2-4acb-a624-4d0cad7bd388-kube-api-access-dtmv6\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.224906 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.224675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.224906 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.224716 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.224906 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.224765 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.224906 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.224809 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.224906 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.224860 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.225217 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.224918 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.225217 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.224939 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.225217 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.224965 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/506453e5-4dc2-4acb-a624-4d0cad7bd388-config-out\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.225217 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.224996 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.225217 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.225017 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.225217 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.225046 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.225217 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.225064 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.325996 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.325965 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-config\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.325996 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.326001 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.326240 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.326019 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/506453e5-4dc2-4acb-a624-4d0cad7bd388-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.326240 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.326037 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.326240 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.326084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-web-config\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.326240 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.326099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.326240 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.326115 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtmv6\" (UniqueName: \"kubernetes.io/projected/506453e5-4dc2-4acb-a624-4d0cad7bd388-kube-api-access-dtmv6\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.326240 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.326140 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.326240 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.326164 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.326240 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.326190 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.326240 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.326213 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.326240 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.326238 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.326586 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.326269 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.326586 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.326293 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.326586 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.326315 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/506453e5-4dc2-4acb-a624-4d0cad7bd388-config-out\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.326586 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.326344 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.326586 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.326366 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.326586 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.326384 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.326586 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.326408 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.326586 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.326458 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.327183 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.327041 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.327285 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.327208 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.328869 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.327848 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.328869 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.328083 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.328869 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.328159 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.328869 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.328497 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.328869 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.328810 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.329944 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.329920 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/506453e5-4dc2-4acb-a624-4d0cad7bd388-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.330489 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.330033 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-web-config\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.330489 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.330041 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.330489 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.330454 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-config\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.331186 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.331160 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.331418 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.331396 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.331681 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.331655 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.331780 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.331748 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/506453e5-4dc2-4acb-a624-4d0cad7bd388-config-out\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.331992 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.331965 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.332138 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.332119 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.332615 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.332593 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.332777 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.332761 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.336496 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.336476 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:54.342955 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.342934 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtmv6\" (UniqueName: \"kubernetes.io/projected/506453e5-4dc2-4acb-a624-4d0cad7bd388-kube-api-access-dtmv6\") pod \"prometheus-k8s-0\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.387009 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.386983 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:54.481078 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.481046 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:54.677946 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.677916 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 02:31:54.680961 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:31:54.680930 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506453e5_4dc2_4acb_a624_4d0cad7bd388.slice/crio-ac707c897c0e903228ae78e5b9d0dfcba53fcb93d595d4a056e2e375bcbc029e WatchSource:0}: Error finding container ac707c897c0e903228ae78e5b9d0dfcba53fcb93d595d4a056e2e375bcbc029e: Status 404 returned error can't find the container with id ac707c897c0e903228ae78e5b9d0dfcba53fcb93d595d4a056e2e375bcbc029e Sep 09 02:31:54.811003 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.810910 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:54.915398 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.915367 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:54.969799 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.969770 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" event={"ID":"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0","Type":"ContainerStarted","Data":"dbabe5ce52e0be7a189a93a4ff0ed9d372aa67d893385ed5e9b155055b6db8ff"} Sep 09 02:31:54.971752 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.971726 2577 generic.go:356] "Generic (PLEG): container finished" podID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerID="c1c5fb084a525b506e529842f2a71dd23a7e200997b81e8221e36610c6495d0e" exitCode=0 Sep 09 02:31:54.971907 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.971841 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"506453e5-4dc2-4acb-a624-4d0cad7bd388","Type":"ContainerDied","Data":"c1c5fb084a525b506e529842f2a71dd23a7e200997b81e8221e36610c6495d0e"} Sep 09 02:31:54.971907 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.971899 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"506453e5-4dc2-4acb-a624-4d0cad7bd388","Type":"ContainerStarted","Data":"ac707c897c0e903228ae78e5b9d0dfcba53fcb93d595d4a056e2e375bcbc029e"} Sep 09 02:31:54.977109 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.977087 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4890a884-dbfb-4177-b14b-2a886198ac47","Type":"ContainerStarted","Data":"92b7b5c4c3c101ca90e3bae02f4d3fb005bceace8b140ac8fee41f1f399ba5a7"} Sep 09 02:31:54.977197 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.977143 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4890a884-dbfb-4177-b14b-2a886198ac47","Type":"ContainerStarted","Data":"f770903772613494eeb46fcc5a14cba734283da774b02073a567a4ba19f6fd4a"} Sep 09 02:31:54.977197 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:54.977170 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4890a884-dbfb-4177-b14b-2a886198ac47","Type":"ContainerStarted","Data":"e82ceecea890975b0b070d3fa6ece64668677fa7cef2b515b34a0b748a896cdd"} Sep 09 02:31:55.056655 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:55.056595 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.289592766 podStartE2EDuration="7.056574015s" totalImagesPullingTime="4.746805998s" totalInitContainerRuntime="20.175251ms" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:55.054236784 +0000 UTC m=+103.184478881" watchObservedRunningTime="2026-09-09 02:31:55.056574015 +0000 UTC m=+103.186816113" Sep 09 02:31:55.988958 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:55.988866 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" event={"ID":"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0","Type":"ContainerStarted","Data":"ee0df38daa9e0a8485efd43b9796e826c2175b4eb2e5d8c2de33f22f209ce6ea"} Sep 09 02:31:55.989404 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:55.988979 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:31:55.989404 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:55.989021 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" event={"ID":"15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0","Type":"ContainerStarted","Data":"62c9760e9ea6381af1ba85d8ca11b16b368d980d0574054474fe007499af15fa"} Sep 09 02:31:55.990393 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:55.990370 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-g26j9" event={"ID":"b01cd589-629e-4438-b7c0-758160094ef9","Type":"ContainerStarted","Data":"c5a08387c5fd2a246fbaf81621488a613a137ba6480978578d4467804b921c7c"} Sep 09 02:31:55.990778 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:55.990761 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-g26j9" Sep 09 02:31:55.995861 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:55.995839 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-g26j9" Sep 09 02:31:56.026333 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:56.026293 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" podStartSLOduration=3.576483123 podStartE2EDuration="7.026281826s" totalImagesPullingTime="3.449798703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:56.025267163 +0000 UTC m=+104.155509260" watchObservedRunningTime="2026-09-09 02:31:56.026281826 +0000 UTC m=+104.156523921" Sep 09 02:31:56.055549 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:56.055503 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-g26j9" podStartSLOduration=2.5576714750000002 podStartE2EDuration="4.055484571s" totalImagesPullingTime="1.497813096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:56.054732577 +0000 UTC m=+104.184974667" watchObservedRunningTime="2026-09-09 02:31:56.055484571 +0000 UTC m=+104.185726669" Sep 09 02:31:56.718902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:56.718873 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:56.814575 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:56.814534 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:56.913620 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:56.913585 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:58.198007 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:58.197978 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:58.286482 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:58.286448 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:31:59.012214 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.012124 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"506453e5-4dc2-4acb-a624-4d0cad7bd388","Type":"ContainerStarted","Data":"f88a0e8931d28b358fe079835a5ff2d83f4e78462d83b8d5f2629897d6becda4"} Sep 09 02:31:59.012214 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.012207 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"506453e5-4dc2-4acb-a624-4d0cad7bd388","Type":"ContainerStarted","Data":"ce2fbe1e33ef3624ce696a0de2ce445893e32375d5a3a95f65115ff80a1cdb30"} Sep 09 02:31:59.012436 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.012234 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"506453e5-4dc2-4acb-a624-4d0cad7bd388","Type":"ContainerStarted","Data":"3e58ac03cc0ad1de0fbb5ca2e99a613b0c2d83bf9b50d72b69b79cfca1d6b696"} Sep 09 02:31:59.012436 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.012258 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"506453e5-4dc2-4acb-a624-4d0cad7bd388","Type":"ContainerStarted","Data":"474b5638ef5b5010c100d4a19060fbeb8d48b08ff266252ece3eec12a1ad415b"} Sep 09 02:31:59.012436 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.012281 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"506453e5-4dc2-4acb-a624-4d0cad7bd388","Type":"ContainerStarted","Data":"239fae9ef6559236f9104bc289129207a19ebfe6afdca958c2335f8c87e167d8"} Sep 09 02:31:59.012436 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.012305 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"506453e5-4dc2-4acb-a624-4d0cad7bd388","Type":"ContainerStarted","Data":"d8e874c5c44b710e695e2cea211c5e37baaa80d79e430db1b5219467b624e449"} Sep 09 02:31:59.055612 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.055561 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.627785255 podStartE2EDuration="5.055545931s" totalImagesPullingTime="3.400787004s" totalInitContainerRuntime="26.973672ms" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:59.053678898 +0000 UTC m=+107.183920993" watchObservedRunningTime="2026-09-09 02:31:59.055545931 +0000 UTC m=+107.185788076" Sep 09 02:31:59.164659 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.164614 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" podUID="e6036480-02a7-47f9-96c0-fcbfefa2d722" containerName="registry" containerID="cri-o://a637bca61b5779a5103b651f2c1000d784df76ac626b27325195fe6c5a86cb95" gracePeriod=30 Sep 09 02:31:59.387175 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.387155 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:31:59.403094 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.403077 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:31:59.475825 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.475794 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-tls\") pod \"e6036480-02a7-47f9-96c0-fcbfefa2d722\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " Sep 09 02:31:59.475986 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.475867 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-certificates\") pod \"e6036480-02a7-47f9-96c0-fcbfefa2d722\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " Sep 09 02:31:59.475986 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.475889 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e6036480-02a7-47f9-96c0-fcbfefa2d722-installation-pull-secrets\") pod \"e6036480-02a7-47f9-96c0-fcbfefa2d722\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " Sep 09 02:31:59.475986 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.475916 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e6036480-02a7-47f9-96c0-fcbfefa2d722-image-registry-private-configuration\") pod \"e6036480-02a7-47f9-96c0-fcbfefa2d722\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " Sep 09 02:31:59.475986 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.475935 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-bound-sa-token\") pod \"e6036480-02a7-47f9-96c0-fcbfefa2d722\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " Sep 09 02:31:59.476194 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.476042 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lmf8\" (UniqueName: \"kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-kube-api-access-4lmf8\") pod \"e6036480-02a7-47f9-96c0-fcbfefa2d722\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " Sep 09 02:31:59.476194 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.476096 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6036480-02a7-47f9-96c0-fcbfefa2d722-trusted-ca\") pod \"e6036480-02a7-47f9-96c0-fcbfefa2d722\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " Sep 09 02:31:59.476194 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.476147 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e6036480-02a7-47f9-96c0-fcbfefa2d722-ca-trust-extracted\") pod \"e6036480-02a7-47f9-96c0-fcbfefa2d722\" (UID: \"e6036480-02a7-47f9-96c0-fcbfefa2d722\") " Sep 09 02:31:59.476429 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.476402 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e6036480-02a7-47f9-96c0-fcbfefa2d722" (UID: "e6036480-02a7-47f9-96c0-fcbfefa2d722"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:31:59.476498 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.476467 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6036480-02a7-47f9-96c0-fcbfefa2d722-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e6036480-02a7-47f9-96c0-fcbfefa2d722" (UID: "e6036480-02a7-47f9-96c0-fcbfefa2d722"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:31:59.476588 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.476564 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6036480-02a7-47f9-96c0-fcbfefa2d722-trusted-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:31:59.476654 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.476592 2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-certificates\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:31:59.478606 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.478570 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6036480-02a7-47f9-96c0-fcbfefa2d722-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e6036480-02a7-47f9-96c0-fcbfefa2d722" (UID: "e6036480-02a7-47f9-96c0-fcbfefa2d722"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:31:59.478711 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.478684 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e6036480-02a7-47f9-96c0-fcbfefa2d722" (UID: "e6036480-02a7-47f9-96c0-fcbfefa2d722"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:31:59.478791 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.478768 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e6036480-02a7-47f9-96c0-fcbfefa2d722" (UID: "e6036480-02a7-47f9-96c0-fcbfefa2d722"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:31:59.478933 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.478911 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6036480-02a7-47f9-96c0-fcbfefa2d722-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e6036480-02a7-47f9-96c0-fcbfefa2d722" (UID: "e6036480-02a7-47f9-96c0-fcbfefa2d722"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:31:59.479106 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.479082 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-kube-api-access-4lmf8" (OuterVolumeSpecName: "kube-api-access-4lmf8") pod "e6036480-02a7-47f9-96c0-fcbfefa2d722" (UID: "e6036480-02a7-47f9-96c0-fcbfefa2d722"). InnerVolumeSpecName "kube-api-access-4lmf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:31:59.484561 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.484538 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6036480-02a7-47f9-96c0-fcbfefa2d722-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e6036480-02a7-47f9-96c0-fcbfefa2d722" (UID: "e6036480-02a7-47f9-96c0-fcbfefa2d722"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:31:59.577384 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.577322 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lmf8\" (UniqueName: \"kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-kube-api-access-4lmf8\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:31:59.577384 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.577348 2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e6036480-02a7-47f9-96c0-fcbfefa2d722-ca-trust-extracted\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:31:59.577384 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.577359 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-registry-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:31:59.577384 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.577367 2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e6036480-02a7-47f9-96c0-fcbfefa2d722-installation-pull-secrets\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:31:59.577384 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.577377 2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e6036480-02a7-47f9-96c0-fcbfefa2d722-image-registry-private-configuration\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:31:59.577384 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:31:59.577385 2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6036480-02a7-47f9-96c0-fcbfefa2d722-bound-sa-token\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:32:00.017064 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:00.017032 2577 generic.go:356] "Generic (PLEG): container finished" podID="e6036480-02a7-47f9-96c0-fcbfefa2d722" containerID="a637bca61b5779a5103b651f2c1000d784df76ac626b27325195fe6c5a86cb95" exitCode=0 Sep 09 02:32:00.017227 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:00.017076 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" Sep 09 02:32:00.017227 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:00.017119 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" event={"ID":"e6036480-02a7-47f9-96c0-fcbfefa2d722","Type":"ContainerDied","Data":"a637bca61b5779a5103b651f2c1000d784df76ac626b27325195fe6c5a86cb95"} Sep 09 02:32:00.017227 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:00.017167 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-df69fdc44-gjkh5" event={"ID":"e6036480-02a7-47f9-96c0-fcbfefa2d722","Type":"ContainerDied","Data":"6ac490240db4122c866ab34c453eac56a880f2ff1a60b1410bc52ed7f575a415"} Sep 09 02:32:00.017227 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:00.017195 2577 scope.go:117] "RemoveContainer" containerID="a637bca61b5779a5103b651f2c1000d784df76ac626b27325195fe6c5a86cb95" Sep 09 02:32:00.026108 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:00.026090 2577 scope.go:117] "RemoveContainer" containerID="a637bca61b5779a5103b651f2c1000d784df76ac626b27325195fe6c5a86cb95" Sep 09 02:32:00.026389 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:32:00.026368 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a637bca61b5779a5103b651f2c1000d784df76ac626b27325195fe6c5a86cb95\": container with ID starting with a637bca61b5779a5103b651f2c1000d784df76ac626b27325195fe6c5a86cb95 not found: ID does not exist" containerID="a637bca61b5779a5103b651f2c1000d784df76ac626b27325195fe6c5a86cb95" Sep 09 02:32:00.026441 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:00.026402 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a637bca61b5779a5103b651f2c1000d784df76ac626b27325195fe6c5a86cb95"} err="failed to get container status \"a637bca61b5779a5103b651f2c1000d784df76ac626b27325195fe6c5a86cb95\": rpc error: code = NotFound desc = could not find container \"a637bca61b5779a5103b651f2c1000d784df76ac626b27325195fe6c5a86cb95\": container with ID starting with a637bca61b5779a5103b651f2c1000d784df76ac626b27325195fe6c5a86cb95 not found: ID does not exist" Sep 09 02:32:00.039564 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:00.039540 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-df69fdc44-gjkh5"] Sep 09 02:32:00.044271 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:00.044252 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-df69fdc44-gjkh5"] Sep 09 02:32:00.309158 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:00.309085 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6036480-02a7-47f9-96c0-fcbfefa2d722" path="/var/lib/kubelet/pods/e6036480-02a7-47f9-96c0-fcbfefa2d722/volumes" Sep 09 02:32:02.001147 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:02.001118 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5cb9bb7cb4-mfqnf" Sep 09 02:32:02.306737 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:02.306660 2577 scope.go:117] "RemoveContainer" containerID="d608c8da21f88bf53025da7ab8fe563de02071d75452695a14973fc730ebcc97" Sep 09 02:32:03.034704 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:03.034672 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/2.log" Sep 09 02:32:03.035247 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:03.034778 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-g849c" event={"ID":"bbed170a-4fc8-4c57-a0dd-320c37ea878c","Type":"ContainerStarted","Data":"923649dcfdb22815d426c22a675d15757495a824485d72404d896ae628a728c4"} Sep 09 02:32:03.035247 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:03.035130 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-g849c" Sep 09 02:32:03.040317 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:03.040296 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-g849c" Sep 09 02:32:03.101680 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:03.101628 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-g849c" podStartSLOduration=56.09609289 podStartE2EDuration="59.101611929s" totalImagesPullingTime="3.005519039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:32:03.100620992 +0000 UTC m=+111.230863089" watchObservedRunningTime="2026-09-09 02:32:03.101611929 +0000 UTC m=+111.231854024" Sep 09 02:32:03.173350 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:03.173313 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:03.263375 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:03.263341 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:03.458593 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:03.458554 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:03.559494 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:03.559463 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:03.641379 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:03.641349 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:03.767173 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:03.767096 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:03.951756 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:03.951723 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:04.029878 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:04.029780 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:04.996241 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:04.996216 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:05.090367 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:05.090339 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:05.191517 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:05.191489 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:05.263396 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:05.263325 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:08.213860 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:08.213808 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:08.312693 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:08.312659 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:08.890967 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:08.890934 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:08.993119 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:08.993084 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:12.780597 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:12.780557 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:12.853257 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:12.853221 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:13.181016 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:13.180977 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:13.262064 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:13.262022 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:15.587098 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:15.587066 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:18.798900 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:18.798863 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:18.905744 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:18.905713 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:19.000413 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:19.000382 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:19.067882 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:19.067777 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:21.162812 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:21.162775 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs\") pod \"network-metrics-daemon-n7lh6\" (UID: \"59a52c60-f713-4a2f-8de1-5d7a2d43ac41\") " pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:32:21.165194 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:21.165169 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a52c60-f713-4a2f-8de1-5d7a2d43ac41-metrics-certs\") pod \"network-metrics-daemon-n7lh6\" (UID: \"59a52c60-f713-4a2f-8de1-5d7a2d43ac41\") " pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:32:21.227167 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:21.227134 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-shl9x\"" Sep 09 02:32:21.234469 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:21.234435 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7lh6" Sep 09 02:32:21.507531 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:21.507506 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n7lh6"] Sep 09 02:32:21.509902 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:32:21.509876 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a52c60_f713_4a2f_8de1_5d7a2d43ac41.slice/crio-efc9a6dd122819c1d5c24720f6e34fab4af64571f33cb43310bc0572e1c5e90e WatchSource:0}: Error finding container efc9a6dd122819c1d5c24720f6e34fab4af64571f33cb43310bc0572e1c5e90e: Status 404 returned error can't find the container with id efc9a6dd122819c1d5c24720f6e34fab4af64571f33cb43310bc0572e1c5e90e Sep 09 02:32:22.114444 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:22.114388 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n7lh6" event={"ID":"59a52c60-f713-4a2f-8de1-5d7a2d43ac41","Type":"ContainerStarted","Data":"efc9a6dd122819c1d5c24720f6e34fab4af64571f33cb43310bc0572e1c5e90e"} Sep 09 02:32:23.120962 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:23.120925 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n7lh6" event={"ID":"59a52c60-f713-4a2f-8de1-5d7a2d43ac41","Type":"ContainerStarted","Data":"f03bf9fe4c9cfb1b2d5c32b0183c62231e2b7538ba47bd545ece0606537079f1"} Sep 09 02:32:23.121325 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:23.120983 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n7lh6" event={"ID":"59a52c60-f713-4a2f-8de1-5d7a2d43ac41","Type":"ContainerStarted","Data":"4cf700757b7a0ce7d75923e1a6cfbb739c80248b73b3b056877726338dc2eb85"} Sep 09 02:32:23.138943 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:23.138873 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-n7lh6" podStartSLOduration=129.165656647 podStartE2EDuration="2m10.138843826s" totalImagesPullingTime="973.187179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:32:23.138588268 +0000 UTC m=+131.268830365" watchObservedRunningTime="2026-09-09 02:32:23.138843826 +0000 UTC m=+131.269085925" Sep 09 02:32:23.263352 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:23.263320 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:23.364113 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:23.364077 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:23.464128 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:23.464101 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:28.182179 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:28.182145 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:28.259042 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:28.259014 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:32.623257 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:32.623220 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:32.695875 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:32.695840 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:33.428203 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:33.428173 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:33.520582 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:33.520546 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:35.426912 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:35.426881 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:35.498403 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:35.498371 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:54.387305 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:54.387267 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:32:54.406219 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:54.406192 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:32:55.266720 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:55.266694 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:32:58.188066 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:58.188029 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:32:58.281635 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:32:58.281607 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:33:07.055677 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:07.055576 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:33:07.158636 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:07.158603 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:33:08.100727 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:08.100686 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 02:33:08.101141 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:08.100994 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="alertmanager" containerID="cri-o://075feaecc46198cedab3ffe885a171402398eca77cc1efedbe6e7657f321bef9" gracePeriod=120 Sep 09 02:33:08.101141 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:08.101070 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="kube-rbac-proxy-metric" containerID="cri-o://f770903772613494eeb46fcc5a14cba734283da774b02073a567a4ba19f6fd4a" gracePeriod=120 Sep 09 02:33:08.101141 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:08.101088 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="kube-rbac-proxy-web" containerID="cri-o://5c364595689175bab0e9d4af3bd7764ec469a0265f152167d09293428de9c432" gracePeriod=120 Sep 09 02:33:08.101141 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:08.101116 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="kube-rbac-proxy" containerID="cri-o://e82ceecea890975b0b070d3fa6ece64668677fa7cef2b515b34a0b748a896cdd" gracePeriod=120 Sep 09 02:33:08.101369 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:08.101143 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="config-reloader" containerID="cri-o://20c543eb80bdac022269b3fdc08bfd17ab8afa0b22d71d26e328a9316e638afd" gracePeriod=120 Sep 09 02:33:08.101369 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:08.101192 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="prom-label-proxy" containerID="cri-o://92b7b5c4c3c101ca90e3bae02f4d3fb005bceace8b140ac8fee41f1f399ba5a7" gracePeriod=120 Sep 09 02:33:08.305566 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:08.305532 2577 generic.go:356] "Generic (PLEG): container finished" podID="4890a884-dbfb-4177-b14b-2a886198ac47" containerID="92b7b5c4c3c101ca90e3bae02f4d3fb005bceace8b140ac8fee41f1f399ba5a7" exitCode=0 Sep 09 02:33:08.305566 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:08.305560 2577 generic.go:356] "Generic (PLEG): container finished" podID="4890a884-dbfb-4177-b14b-2a886198ac47" containerID="f770903772613494eeb46fcc5a14cba734283da774b02073a567a4ba19f6fd4a" exitCode=0 Sep 09 02:33:08.305566 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:08.305571 2577 generic.go:356] "Generic (PLEG): container finished" podID="4890a884-dbfb-4177-b14b-2a886198ac47" containerID="e82ceecea890975b0b070d3fa6ece64668677fa7cef2b515b34a0b748a896cdd" exitCode=0 Sep 09 02:33:08.305767 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:08.305579 2577 generic.go:356] "Generic (PLEG): container finished" podID="4890a884-dbfb-4177-b14b-2a886198ac47" containerID="20c543eb80bdac022269b3fdc08bfd17ab8afa0b22d71d26e328a9316e638afd" exitCode=0 Sep 09 02:33:08.305767 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:08.305588 2577 generic.go:356] "Generic (PLEG): container finished" podID="4890a884-dbfb-4177-b14b-2a886198ac47" containerID="075feaecc46198cedab3ffe885a171402398eca77cc1efedbe6e7657f321bef9" exitCode=0 Sep 09 02:33:08.308748 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:08.308720 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4890a884-dbfb-4177-b14b-2a886198ac47","Type":"ContainerDied","Data":"92b7b5c4c3c101ca90e3bae02f4d3fb005bceace8b140ac8fee41f1f399ba5a7"} Sep 09 02:33:08.308878 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:08.308793 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4890a884-dbfb-4177-b14b-2a886198ac47","Type":"ContainerDied","Data":"f770903772613494eeb46fcc5a14cba734283da774b02073a567a4ba19f6fd4a"} Sep 09 02:33:08.308878 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:08.308855 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4890a884-dbfb-4177-b14b-2a886198ac47","Type":"ContainerDied","Data":"e82ceecea890975b0b070d3fa6ece64668677fa7cef2b515b34a0b748a896cdd"} Sep 09 02:33:08.308989 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:08.308893 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4890a884-dbfb-4177-b14b-2a886198ac47","Type":"ContainerDied","Data":"20c543eb80bdac022269b3fdc08bfd17ab8afa0b22d71d26e328a9316e638afd"} Sep 09 02:33:08.308989 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:08.308934 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4890a884-dbfb-4177-b14b-2a886198ac47","Type":"ContainerDied","Data":"075feaecc46198cedab3ffe885a171402398eca77cc1efedbe6e7657f321bef9"} Sep 09 02:33:09.316331 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.316297 2577 generic.go:356] "Generic (PLEG): container finished" podID="4890a884-dbfb-4177-b14b-2a886198ac47" containerID="5c364595689175bab0e9d4af3bd7764ec469a0265f152167d09293428de9c432" exitCode=0 Sep 09 02:33:09.316688 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.316362 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4890a884-dbfb-4177-b14b-2a886198ac47","Type":"ContainerDied","Data":"5c364595689175bab0e9d4af3bd7764ec469a0265f152167d09293428de9c432"} Sep 09 02:33:09.347269 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.347248 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:09.371947 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.371919 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4890a884-dbfb-4177-b14b-2a886198ac47-tls-assets\") pod \"4890a884-dbfb-4177-b14b-2a886198ac47\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " Sep 09 02:33:09.372051 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.371963 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4890a884-dbfb-4177-b14b-2a886198ac47-alertmanager-trusted-ca-bundle\") pod \"4890a884-dbfb-4177-b14b-2a886198ac47\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " Sep 09 02:33:09.372051 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.371993 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-main-tls\") pod \"4890a884-dbfb-4177-b14b-2a886198ac47\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " Sep 09 02:33:09.372051 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.372036 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4890a884-dbfb-4177-b14b-2a886198ac47-alertmanager-main-db\") pod \"4890a884-dbfb-4177-b14b-2a886198ac47\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " Sep 09 02:33:09.372215 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.372061 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-web-config\") pod \"4890a884-dbfb-4177-b14b-2a886198ac47\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " Sep 09 02:33:09.372215 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.372158 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4890a884-dbfb-4177-b14b-2a886198ac47-config-out\") pod \"4890a884-dbfb-4177-b14b-2a886198ac47\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " Sep 09 02:33:09.372315 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.372231 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-kube-rbac-proxy\") pod \"4890a884-dbfb-4177-b14b-2a886198ac47\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " Sep 09 02:33:09.372315 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.372272 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4890a884-dbfb-4177-b14b-2a886198ac47-metrics-client-ca\") pod \"4890a884-dbfb-4177-b14b-2a886198ac47\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " Sep 09 02:33:09.372428 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.372314 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-config-volume\") pod \"4890a884-dbfb-4177-b14b-2a886198ac47\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " Sep 09 02:33:09.372428 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.372339 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-kube-rbac-proxy-web\") pod \"4890a884-dbfb-4177-b14b-2a886198ac47\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " Sep 09 02:33:09.372428 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.372350 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4890a884-dbfb-4177-b14b-2a886198ac47-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "4890a884-dbfb-4177-b14b-2a886198ac47" (UID: "4890a884-dbfb-4177-b14b-2a886198ac47"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:33:09.372428 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.372390 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-cluster-tls-config\") pod \"4890a884-dbfb-4177-b14b-2a886198ac47\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " Sep 09 02:33:09.372428 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.372391 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4890a884-dbfb-4177-b14b-2a886198ac47-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "4890a884-dbfb-4177-b14b-2a886198ac47" (UID: "4890a884-dbfb-4177-b14b-2a886198ac47"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:33:09.372428 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.372423 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxlxf\" (UniqueName: \"kubernetes.io/projected/4890a884-dbfb-4177-b14b-2a886198ac47-kube-api-access-cxlxf\") pod \"4890a884-dbfb-4177-b14b-2a886198ac47\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " Sep 09 02:33:09.372762 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.372449 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-kube-rbac-proxy-metric\") pod \"4890a884-dbfb-4177-b14b-2a886198ac47\" (UID: \"4890a884-dbfb-4177-b14b-2a886198ac47\") " Sep 09 02:33:09.372762 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.372707 2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4890a884-dbfb-4177-b14b-2a886198ac47-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:09.372762 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.372726 2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4890a884-dbfb-4177-b14b-2a886198ac47-alertmanager-main-db\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:09.373759 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.373728 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4890a884-dbfb-4177-b14b-2a886198ac47-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "4890a884-dbfb-4177-b14b-2a886198ac47" (UID: "4890a884-dbfb-4177-b14b-2a886198ac47"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:33:09.375148 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.375112 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4890a884-dbfb-4177-b14b-2a886198ac47-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4890a884-dbfb-4177-b14b-2a886198ac47" (UID: "4890a884-dbfb-4177-b14b-2a886198ac47"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:33:09.375511 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.375480 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "4890a884-dbfb-4177-b14b-2a886198ac47" (UID: "4890a884-dbfb-4177-b14b-2a886198ac47"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:33:09.375727 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.375701 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4890a884-dbfb-4177-b14b-2a886198ac47-config-out" (OuterVolumeSpecName: "config-out") pod "4890a884-dbfb-4177-b14b-2a886198ac47" (UID: "4890a884-dbfb-4177-b14b-2a886198ac47"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:33:09.375855 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.375699 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "4890a884-dbfb-4177-b14b-2a886198ac47" (UID: "4890a884-dbfb-4177-b14b-2a886198ac47"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:33:09.376070 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.376036 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "4890a884-dbfb-4177-b14b-2a886198ac47" (UID: "4890a884-dbfb-4177-b14b-2a886198ac47"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:33:09.376483 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.376442 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-config-volume" (OuterVolumeSpecName: "config-volume") pod "4890a884-dbfb-4177-b14b-2a886198ac47" (UID: "4890a884-dbfb-4177-b14b-2a886198ac47"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:33:09.376579 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.376546 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4890a884-dbfb-4177-b14b-2a886198ac47-kube-api-access-cxlxf" (OuterVolumeSpecName: "kube-api-access-cxlxf") pod "4890a884-dbfb-4177-b14b-2a886198ac47" (UID: "4890a884-dbfb-4177-b14b-2a886198ac47"). InnerVolumeSpecName "kube-api-access-cxlxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:33:09.376842 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.376799 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "4890a884-dbfb-4177-b14b-2a886198ac47" (UID: "4890a884-dbfb-4177-b14b-2a886198ac47"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:33:09.380770 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.380687 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "4890a884-dbfb-4177-b14b-2a886198ac47" (UID: "4890a884-dbfb-4177-b14b-2a886198ac47"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:33:09.390393 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.390356 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-web-config" (OuterVolumeSpecName: "web-config") pod "4890a884-dbfb-4177-b14b-2a886198ac47" (UID: "4890a884-dbfb-4177-b14b-2a886198ac47"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:33:09.473681 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.473581 2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-config-volume\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:09.473681 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.473618 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:09.473681 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.473633 2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-cluster-tls-config\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:09.473681 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.473645 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxlxf\" (UniqueName: \"kubernetes.io/projected/4890a884-dbfb-4177-b14b-2a886198ac47-kube-api-access-cxlxf\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:09.473681 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.473658 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:09.473681 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.473670 2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4890a884-dbfb-4177-b14b-2a886198ac47-tls-assets\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:09.473681 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.473682 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-main-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:09.474071 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.473693 2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-web-config\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:09.474071 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.473704 2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4890a884-dbfb-4177-b14b-2a886198ac47-config-out\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:09.474071 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.473714 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4890a884-dbfb-4177-b14b-2a886198ac47-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:09.474071 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:09.473727 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4890a884-dbfb-4177-b14b-2a886198ac47-metrics-client-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:10.323429 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.323403 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4890a884-dbfb-4177-b14b-2a886198ac47","Type":"ContainerDied","Data":"b700819d757c93a868acc61dcb94e4d0cbf6e83efa4c1ab5d38898a9cdd1e2f6"} Sep 09 02:33:10.323842 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.323461 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.323842 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.323469 2577 scope.go:117] "RemoveContainer" containerID="92b7b5c4c3c101ca90e3bae02f4d3fb005bceace8b140ac8fee41f1f399ba5a7" Sep 09 02:33:10.331302 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.331276 2577 scope.go:117] "RemoveContainer" containerID="f770903772613494eeb46fcc5a14cba734283da774b02073a567a4ba19f6fd4a" Sep 09 02:33:10.338176 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.338158 2577 scope.go:117] "RemoveContainer" containerID="e82ceecea890975b0b070d3fa6ece64668677fa7cef2b515b34a0b748a896cdd" Sep 09 02:33:10.344927 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.344908 2577 scope.go:117] "RemoveContainer" containerID="5c364595689175bab0e9d4af3bd7764ec469a0265f152167d09293428de9c432" Sep 09 02:33:10.351559 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.351540 2577 scope.go:117] "RemoveContainer" containerID="20c543eb80bdac022269b3fdc08bfd17ab8afa0b22d71d26e328a9316e638afd" Sep 09 02:33:10.358253 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.358237 2577 scope.go:117] "RemoveContainer" containerID="075feaecc46198cedab3ffe885a171402398eca77cc1efedbe6e7657f321bef9" Sep 09 02:33:10.363682 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.363664 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 02:33:10.366028 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.366011 2577 scope.go:117] "RemoveContainer" containerID="40bd3e30bafeddee487114980193be1f082d702baf11c70e2e33570e4a9ba3d8" Sep 09 02:33:10.376795 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.376775 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 02:33:10.412868 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.412843 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 02:33:10.413198 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413183 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="kube-rbac-proxy-metric" Sep 09 02:33:10.413281 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413202 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="kube-rbac-proxy-metric" Sep 09 02:33:10.413281 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413224 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="config-reloader" Sep 09 02:33:10.413281 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413262 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="config-reloader" Sep 09 02:33:10.413432 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413292 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6036480-02a7-47f9-96c0-fcbfefa2d722" containerName="registry" Sep 09 02:33:10.413432 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413301 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6036480-02a7-47f9-96c0-fcbfefa2d722" containerName="registry" Sep 09 02:33:10.413432 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413310 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="prom-label-proxy" Sep 09 02:33:10.413432 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413319 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="prom-label-proxy" Sep 09 02:33:10.413432 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413330 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="kube-rbac-proxy" Sep 09 02:33:10.413432 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413338 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="kube-rbac-proxy" Sep 09 02:33:10.413432 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413351 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="init-config-reloader" Sep 09 02:33:10.413432 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413359 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="init-config-reloader" Sep 09 02:33:10.413432 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413369 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="kube-rbac-proxy-web" Sep 09 02:33:10.413432 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413378 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="kube-rbac-proxy-web" Sep 09 02:33:10.413432 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413392 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="alertmanager" Sep 09 02:33:10.413432 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413400 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="alertmanager" Sep 09 02:33:10.414033 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413491 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="config-reloader" Sep 09 02:33:10.414033 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413503 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="kube-rbac-proxy" Sep 09 02:33:10.414033 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413513 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="kube-rbac-proxy-web" Sep 09 02:33:10.414033 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413521 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6036480-02a7-47f9-96c0-fcbfefa2d722" containerName="registry" Sep 09 02:33:10.414033 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413531 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="kube-rbac-proxy-metric" Sep 09 02:33:10.414033 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413541 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="alertmanager" Sep 09 02:33:10.414033 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.413549 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" containerName="prom-label-proxy" Sep 09 02:33:10.417310 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.417292 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.420424 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.420391 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 09 02:33:10.420520 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.420407 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 09 02:33:10.420520 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.420459 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 09 02:33:10.420520 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.420472 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 09 02:33:10.420520 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.420507 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 09 02:33:10.420723 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.420581 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 09 02:33:10.420723 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.420620 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 09 02:33:10.420857 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.420844 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 09 02:33:10.421067 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.421054 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ssw27\"" Sep 09 02:33:10.428919 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.428899 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 09 02:33:10.435241 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.435219 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 02:33:10.481874 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.481771 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5fa55c8a-6da0-46e5-b559-e13972dee6e7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.481874 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.481812 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5fa55c8a-6da0-46e5-b559-e13972dee6e7-config-volume\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.481874 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.481862 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5fa55c8a-6da0-46e5-b559-e13972dee6e7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.482082 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.481894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fa55c8a-6da0-46e5-b559-e13972dee6e7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.482082 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.481953 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5fa55c8a-6da0-46e5-b559-e13972dee6e7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.482082 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.481984 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5fa55c8a-6da0-46e5-b559-e13972dee6e7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.482082 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.482005 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fa55c8a-6da0-46e5-b559-e13972dee6e7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.482233 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.482085 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5fa55c8a-6da0-46e5-b559-e13972dee6e7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.482233 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.482111 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pn98\" (UniqueName: \"kubernetes.io/projected/5fa55c8a-6da0-46e5-b559-e13972dee6e7-kube-api-access-8pn98\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.482233 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.482140 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5fa55c8a-6da0-46e5-b559-e13972dee6e7-config-out\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.482233 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.482163 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5fa55c8a-6da0-46e5-b559-e13972dee6e7-web-config\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.482233 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.482178 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5fa55c8a-6da0-46e5-b559-e13972dee6e7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.482233 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.482222 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5fa55c8a-6da0-46e5-b559-e13972dee6e7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.582961 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.582928 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5fa55c8a-6da0-46e5-b559-e13972dee6e7-web-config\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.582961 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.582960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5fa55c8a-6da0-46e5-b559-e13972dee6e7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.583153 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.582981 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5fa55c8a-6da0-46e5-b559-e13972dee6e7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.583153 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.583010 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5fa55c8a-6da0-46e5-b559-e13972dee6e7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.583260 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.583144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5fa55c8a-6da0-46e5-b559-e13972dee6e7-config-volume\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.583260 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.583189 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5fa55c8a-6da0-46e5-b559-e13972dee6e7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.583260 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.583227 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fa55c8a-6da0-46e5-b559-e13972dee6e7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.583407 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.583263 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5fa55c8a-6da0-46e5-b559-e13972dee6e7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.583407 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.583299 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5fa55c8a-6da0-46e5-b559-e13972dee6e7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.583407 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.583326 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fa55c8a-6da0-46e5-b559-e13972dee6e7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.583407 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.583374 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5fa55c8a-6da0-46e5-b559-e13972dee6e7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.583407 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.583399 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5fa55c8a-6da0-46e5-b559-e13972dee6e7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.583626 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.583408 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pn98\" (UniqueName: \"kubernetes.io/projected/5fa55c8a-6da0-46e5-b559-e13972dee6e7-kube-api-access-8pn98\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.583626 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.583474 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5fa55c8a-6da0-46e5-b559-e13972dee6e7-config-out\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.585277 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.584380 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fa55c8a-6da0-46e5-b559-e13972dee6e7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.585277 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.584972 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fa55c8a-6da0-46e5-b559-e13972dee6e7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.586073 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.586041 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5fa55c8a-6da0-46e5-b559-e13972dee6e7-config-out\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.586169 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.586085 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5fa55c8a-6da0-46e5-b559-e13972dee6e7-web-config\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.586278 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.586258 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5fa55c8a-6da0-46e5-b559-e13972dee6e7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.586491 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.586456 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5fa55c8a-6da0-46e5-b559-e13972dee6e7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.586585 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.586463 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5fa55c8a-6da0-46e5-b559-e13972dee6e7-config-volume\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.586585 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.586529 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5fa55c8a-6da0-46e5-b559-e13972dee6e7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.586757 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.586738 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5fa55c8a-6da0-46e5-b559-e13972dee6e7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.587670 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.587647 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5fa55c8a-6da0-46e5-b559-e13972dee6e7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.588141 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.588126 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5fa55c8a-6da0-46e5-b559-e13972dee6e7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.592338 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.592318 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pn98\" (UniqueName: \"kubernetes.io/projected/5fa55c8a-6da0-46e5-b559-e13972dee6e7-kube-api-access-8pn98\") pod \"alertmanager-main-0\" (UID: \"5fa55c8a-6da0-46e5-b559-e13972dee6e7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:10.726490 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:10.726452 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 02:33:11.018802 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:11.018779 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 02:33:11.020919 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:33:11.020892 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa55c8a_6da0_46e5_b559_e13972dee6e7.slice/crio-f5618bb290656534a3cdf81986a4c67ce79b2acaca30df921576b9b098f80f04 WatchSource:0}: Error finding container f5618bb290656534a3cdf81986a4c67ce79b2acaca30df921576b9b098f80f04: Status 404 returned error can't find the container with id f5618bb290656534a3cdf81986a4c67ce79b2acaca30df921576b9b098f80f04 Sep 09 02:33:11.330687 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:11.330605 2577 generic.go:356] "Generic (PLEG): container finished" podID="5fa55c8a-6da0-46e5-b559-e13972dee6e7" containerID="15f6475f3c01c5a9c2127ff900a5520e7e4624cc9ee72dfbcafbb7360a2bea6b" exitCode=0 Sep 09 02:33:11.331128 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:11.330696 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fa55c8a-6da0-46e5-b559-e13972dee6e7","Type":"ContainerDied","Data":"15f6475f3c01c5a9c2127ff900a5520e7e4624cc9ee72dfbcafbb7360a2bea6b"} Sep 09 02:33:11.331128 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:11.330772 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fa55c8a-6da0-46e5-b559-e13972dee6e7","Type":"ContainerStarted","Data":"f5618bb290656534a3cdf81986a4c67ce79b2acaca30df921576b9b098f80f04"} Sep 09 02:33:12.176623 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.176588 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6"] Sep 09 02:33:12.180226 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.180208 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.183776 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.183753 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 09 02:33:12.183927 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.183836 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 09 02:33:12.184267 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.184251 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 09 02:33:12.184458 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.184443 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 09 02:33:12.184550 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.184445 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-q476k\"" Sep 09 02:33:12.185058 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.185041 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 09 02:33:12.189323 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.189304 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 09 02:33:12.196882 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.196862 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6"] Sep 09 02:33:12.297680 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.297657 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ea9506c8-63c6-4868-b855-d938d9858e35-federate-client-tls\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.297791 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.297687 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ea9506c8-63c6-4868-b855-d938d9858e35-secret-telemeter-client\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.297791 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.297706 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx9fh\" (UniqueName: \"kubernetes.io/projected/ea9506c8-63c6-4868-b855-d938d9858e35-kube-api-access-dx9fh\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.297791 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.297737 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea9506c8-63c6-4868-b855-d938d9858e35-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.297791 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.297777 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea9506c8-63c6-4868-b855-d938d9858e35-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.298039 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.297834 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea9506c8-63c6-4868-b855-d938d9858e35-serving-certs-ca-bundle\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.298039 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.297872 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ea9506c8-63c6-4868-b855-d938d9858e35-telemeter-client-tls\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.298039 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.297971 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea9506c8-63c6-4868-b855-d938d9858e35-metrics-client-ca\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.308496 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.308465 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4890a884-dbfb-4177-b14b-2a886198ac47" path="/var/lib/kubelet/pods/4890a884-dbfb-4177-b14b-2a886198ac47/volumes" Sep 09 02:33:12.338757 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.338733 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fa55c8a-6da0-46e5-b559-e13972dee6e7","Type":"ContainerStarted","Data":"278b0269b11baa427e24275c6af91f86dffe38f8b222fa7564bc5d2df39534d5"} Sep 09 02:33:12.339158 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.338800 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fa55c8a-6da0-46e5-b559-e13972dee6e7","Type":"ContainerStarted","Data":"b2f2aaa3aa9b8fd7889c1fa0ffad32d8e7ddcf1f0a90eec2a7f3fc11a8f60608"} Sep 09 02:33:12.339158 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.338875 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fa55c8a-6da0-46e5-b559-e13972dee6e7","Type":"ContainerStarted","Data":"6613db1facae70ba496bdf9c13a4559042e4a662f62b4b4d32cc8646496a5b2a"} Sep 09 02:33:12.339158 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.338917 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fa55c8a-6da0-46e5-b559-e13972dee6e7","Type":"ContainerStarted","Data":"83be686a4ca5770a274d6dfa23a329dff30ca5242f0a3b9a823c0e8513f9dfc5"} Sep 09 02:33:12.339158 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.338964 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fa55c8a-6da0-46e5-b559-e13972dee6e7","Type":"ContainerStarted","Data":"12449b8a11d5801e525cd3d7ecf098d3de1e6f553aee38f9d90707f8daf2c7d4"} Sep 09 02:33:12.339158 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.339004 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fa55c8a-6da0-46e5-b559-e13972dee6e7","Type":"ContainerStarted","Data":"dd8784f712895bfea4338ac285280830a05ebe8e1f101342850fe3c4662b4f93"} Sep 09 02:33:12.371823 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.371781 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.356310171 podStartE2EDuration="2.371763632s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.453461ms" isStatefulPod=false podCreationTimestamp="2026-09-09 02:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:33:12.369715368 +0000 UTC m=+180.499957491" watchObservedRunningTime="2026-09-09 02:33:12.371763632 +0000 UTC m=+180.502005728" Sep 09 02:33:12.399457 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.399418 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx9fh\" (UniqueName: \"kubernetes.io/projected/ea9506c8-63c6-4868-b855-d938d9858e35-kube-api-access-dx9fh\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.399583 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.399522 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea9506c8-63c6-4868-b855-d938d9858e35-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.399583 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.399558 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea9506c8-63c6-4868-b855-d938d9858e35-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.399687 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.399620 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea9506c8-63c6-4868-b855-d938d9858e35-serving-certs-ca-bundle\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.399772 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.399706 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ea9506c8-63c6-4868-b855-d938d9858e35-telemeter-client-tls\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.399906 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.399884 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea9506c8-63c6-4868-b855-d938d9858e35-metrics-client-ca\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.399982 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.399971 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ea9506c8-63c6-4868-b855-d938d9858e35-federate-client-tls\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.400041 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.400023 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ea9506c8-63c6-4868-b855-d938d9858e35-secret-telemeter-client\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.401221 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.401198 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea9506c8-63c6-4868-b855-d938d9858e35-metrics-client-ca\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.405851 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.405829 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 09 02:33:12.405950 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.405832 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 09 02:33:12.405950 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.405940 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 09 02:33:12.406079 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.405956 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 09 02:33:12.406079 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.405966 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 09 02:33:12.410466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.410361 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea9506c8-63c6-4868-b855-d938d9858e35-serving-certs-ca-bundle\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.411695 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.411522 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 09 02:33:12.412533 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.412511 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx9fh\" (UniqueName: \"kubernetes.io/projected/ea9506c8-63c6-4868-b855-d938d9858e35-kube-api-access-dx9fh\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.412533 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.412525 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ea9506c8-63c6-4868-b855-d938d9858e35-telemeter-client-tls\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.412693 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.412630 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ea9506c8-63c6-4868-b855-d938d9858e35-secret-telemeter-client\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.413044 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.413020 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea9506c8-63c6-4868-b855-d938d9858e35-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.413085 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.413071 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ea9506c8-63c6-4868-b855-d938d9858e35-federate-client-tls\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.421312 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.421293 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea9506c8-63c6-4868-b855-d938d9858e35-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54b8dcbdd-9zrn6\" (UID: \"ea9506c8-63c6-4868-b855-d938d9858e35\") " pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.474602 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.474532 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 02:33:12.474865 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.474838 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="kube-rbac-proxy" containerID="cri-o://ce2fbe1e33ef3624ce696a0de2ce445893e32375d5a3a95f65115ff80a1cdb30" gracePeriod=600 Sep 09 02:33:12.475019 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.474835 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="thanos-sidecar" containerID="cri-o://474b5638ef5b5010c100d4a19060fbeb8d48b08ff266252ece3eec12a1ad415b" gracePeriod=600 Sep 09 02:33:12.475145 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.474812 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="prometheus" containerID="cri-o://d8e874c5c44b710e695e2cea211c5e37baaa80d79e430db1b5219467b624e449" gracePeriod=600 Sep 09 02:33:12.475209 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.474863 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="config-reloader" containerID="cri-o://239fae9ef6559236f9104bc289129207a19ebfe6afdca958c2335f8c87e167d8" gracePeriod=600 Sep 09 02:33:12.475254 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.474861 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="kube-rbac-proxy-web" containerID="cri-o://3e58ac03cc0ad1de0fbb5ca2e99a613b0c2d83bf9b50d72b69b79cfca1d6b696" gracePeriod=600 Sep 09 02:33:12.475899 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.474876 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f88a0e8931d28b358fe079835a5ff2d83f4e78462d83b8d5f2629897d6becda4" gracePeriod=600 Sep 09 02:33:12.496908 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.495529 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-q476k\"" Sep 09 02:33:12.500329 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.500310 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" Sep 09 02:33:12.785073 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:12.785044 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6"] Sep 09 02:33:12.787461 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:33:12.787437 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea9506c8_63c6_4868_b855_d938d9858e35.slice/crio-389c4e2b4f9b47ba8cda8e7371b1eb594d7b7eee67fa79b0e06216cffe822560 WatchSource:0}: Error finding container 389c4e2b4f9b47ba8cda8e7371b1eb594d7b7eee67fa79b0e06216cffe822560: Status 404 returned error can't find the container with id 389c4e2b4f9b47ba8cda8e7371b1eb594d7b7eee67fa79b0e06216cffe822560 Sep 09 02:33:13.348272 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.348239 2577 generic.go:356] "Generic (PLEG): container finished" podID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerID="f88a0e8931d28b358fe079835a5ff2d83f4e78462d83b8d5f2629897d6becda4" exitCode=0 Sep 09 02:33:13.348272 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.348267 2577 generic.go:356] "Generic (PLEG): container finished" podID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerID="ce2fbe1e33ef3624ce696a0de2ce445893e32375d5a3a95f65115ff80a1cdb30" exitCode=0 Sep 09 02:33:13.348272 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.348276 2577 generic.go:356] "Generic (PLEG): container finished" podID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerID="474b5638ef5b5010c100d4a19060fbeb8d48b08ff266252ece3eec12a1ad415b" exitCode=0 Sep 09 02:33:13.348272 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.348282 2577 generic.go:356] "Generic (PLEG): container finished" podID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerID="239fae9ef6559236f9104bc289129207a19ebfe6afdca958c2335f8c87e167d8" exitCode=0 Sep 09 02:33:13.348839 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.348287 2577 generic.go:356] "Generic (PLEG): container finished" podID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerID="d8e874c5c44b710e695e2cea211c5e37baaa80d79e430db1b5219467b624e449" exitCode=0 Sep 09 02:33:13.348839 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.348314 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"506453e5-4dc2-4acb-a624-4d0cad7bd388","Type":"ContainerDied","Data":"f88a0e8931d28b358fe079835a5ff2d83f4e78462d83b8d5f2629897d6becda4"} Sep 09 02:33:13.348839 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.348402 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"506453e5-4dc2-4acb-a624-4d0cad7bd388","Type":"ContainerDied","Data":"ce2fbe1e33ef3624ce696a0de2ce445893e32375d5a3a95f65115ff80a1cdb30"} Sep 09 02:33:13.348839 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.348450 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"506453e5-4dc2-4acb-a624-4d0cad7bd388","Type":"ContainerDied","Data":"474b5638ef5b5010c100d4a19060fbeb8d48b08ff266252ece3eec12a1ad415b"} Sep 09 02:33:13.348839 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.348490 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"506453e5-4dc2-4acb-a624-4d0cad7bd388","Type":"ContainerDied","Data":"239fae9ef6559236f9104bc289129207a19ebfe6afdca958c2335f8c87e167d8"} Sep 09 02:33:13.348839 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.348529 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"506453e5-4dc2-4acb-a624-4d0cad7bd388","Type":"ContainerDied","Data":"d8e874c5c44b710e695e2cea211c5e37baaa80d79e430db1b5219467b624e449"} Sep 09 02:33:13.349803 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.349775 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" event={"ID":"ea9506c8-63c6-4868-b855-d938d9858e35","Type":"ContainerStarted","Data":"389c4e2b4f9b47ba8cda8e7371b1eb594d7b7eee67fa79b0e06216cffe822560"} Sep 09 02:33:13.743438 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.743414 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:13.812913 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.812886 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtmv6\" (UniqueName: \"kubernetes.io/projected/506453e5-4dc2-4acb-a624-4d0cad7bd388-kube-api-access-dtmv6\") pod \"506453e5-4dc2-4acb-a624-4d0cad7bd388\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " Sep 09 02:33:13.813082 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.812924 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-prometheus-k8s-tls\") pod \"506453e5-4dc2-4acb-a624-4d0cad7bd388\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " Sep 09 02:33:13.813082 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.812953 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-db\") pod \"506453e5-4dc2-4acb-a624-4d0cad7bd388\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " Sep 09 02:33:13.813082 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.812979 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-rulefiles-1\") pod \"506453e5-4dc2-4acb-a624-4d0cad7bd388\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " Sep 09 02:33:13.813082 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.813009 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-grpc-tls\") pod \"506453e5-4dc2-4acb-a624-4d0cad7bd388\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " Sep 09 02:33:13.813082 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.813043 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-configmap-metrics-client-ca\") pod \"506453e5-4dc2-4acb-a624-4d0cad7bd388\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " Sep 09 02:33:13.813082 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.813071 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-web-config\") pod \"506453e5-4dc2-4acb-a624-4d0cad7bd388\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " Sep 09 02:33:13.813418 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.813122 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-configmap-serving-certs-ca-bundle\") pod \"506453e5-4dc2-4acb-a624-4d0cad7bd388\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " Sep 09 02:33:13.813418 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.813156 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-trusted-ca-bundle\") pod \"506453e5-4dc2-4acb-a624-4d0cad7bd388\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " Sep 09 02:33:13.813418 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.813189 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-rulefiles-0\") pod \"506453e5-4dc2-4acb-a624-4d0cad7bd388\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " Sep 09 02:33:13.813418 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.813213 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/506453e5-4dc2-4acb-a624-4d0cad7bd388-config-out\") pod \"506453e5-4dc2-4acb-a624-4d0cad7bd388\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " Sep 09 02:33:13.813418 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.813245 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/506453e5-4dc2-4acb-a624-4d0cad7bd388-tls-assets\") pod \"506453e5-4dc2-4acb-a624-4d0cad7bd388\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " Sep 09 02:33:13.813418 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.813274 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-metrics-client-certs\") pod \"506453e5-4dc2-4acb-a624-4d0cad7bd388\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " Sep 09 02:33:13.813418 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.813305 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"506453e5-4dc2-4acb-a624-4d0cad7bd388\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " Sep 09 02:33:13.813418 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.813370 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-configmap-kubelet-serving-ca-bundle\") pod \"506453e5-4dc2-4acb-a624-4d0cad7bd388\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " Sep 09 02:33:13.813418 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.813394 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-thanos-prometheus-http-client-file\") pod \"506453e5-4dc2-4acb-a624-4d0cad7bd388\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " Sep 09 02:33:13.813847 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.813426 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-config\") pod \"506453e5-4dc2-4acb-a624-4d0cad7bd388\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " Sep 09 02:33:13.813847 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.813454 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"506453e5-4dc2-4acb-a624-4d0cad7bd388\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " Sep 09 02:33:13.813847 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.813489 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-rulefiles-2\") pod \"506453e5-4dc2-4acb-a624-4d0cad7bd388\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " Sep 09 02:33:13.813847 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.813524 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-kube-rbac-proxy\") pod \"506453e5-4dc2-4acb-a624-4d0cad7bd388\" (UID: \"506453e5-4dc2-4acb-a624-4d0cad7bd388\") " Sep 09 02:33:13.813847 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.813594 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "506453e5-4dc2-4acb-a624-4d0cad7bd388" (UID: "506453e5-4dc2-4acb-a624-4d0cad7bd388"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:33:13.814135 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.813861 2577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-configmap-metrics-client-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:13.814193 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.814125 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "506453e5-4dc2-4acb-a624-4d0cad7bd388" (UID: "506453e5-4dc2-4acb-a624-4d0cad7bd388"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:33:13.814791 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.814500 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "506453e5-4dc2-4acb-a624-4d0cad7bd388" (UID: "506453e5-4dc2-4acb-a624-4d0cad7bd388"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:33:13.814791 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.814645 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "506453e5-4dc2-4acb-a624-4d0cad7bd388" (UID: "506453e5-4dc2-4acb-a624-4d0cad7bd388"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:33:13.815225 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.815196 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "506453e5-4dc2-4acb-a624-4d0cad7bd388" (UID: "506453e5-4dc2-4acb-a624-4d0cad7bd388"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:33:13.815960 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.815709 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "506453e5-4dc2-4acb-a624-4d0cad7bd388" (UID: "506453e5-4dc2-4acb-a624-4d0cad7bd388"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:33:13.815960 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.815930 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "506453e5-4dc2-4acb-a624-4d0cad7bd388" (UID: "506453e5-4dc2-4acb-a624-4d0cad7bd388"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:33:13.816138 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.816026 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "506453e5-4dc2-4acb-a624-4d0cad7bd388" (UID: "506453e5-4dc2-4acb-a624-4d0cad7bd388"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:33:13.816491 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.816458 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "506453e5-4dc2-4acb-a624-4d0cad7bd388" (UID: "506453e5-4dc2-4acb-a624-4d0cad7bd388"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:33:13.817320 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.817292 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "506453e5-4dc2-4acb-a624-4d0cad7bd388" (UID: "506453e5-4dc2-4acb-a624-4d0cad7bd388"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:33:13.817935 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.817894 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-config" (OuterVolumeSpecName: "config") pod "506453e5-4dc2-4acb-a624-4d0cad7bd388" (UID: "506453e5-4dc2-4acb-a624-4d0cad7bd388"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:33:13.819128 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.819072 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "506453e5-4dc2-4acb-a624-4d0cad7bd388" (UID: "506453e5-4dc2-4acb-a624-4d0cad7bd388"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:33:13.819128 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.819117 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/506453e5-4dc2-4acb-a624-4d0cad7bd388-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "506453e5-4dc2-4acb-a624-4d0cad7bd388" (UID: "506453e5-4dc2-4acb-a624-4d0cad7bd388"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:33:13.819255 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.819237 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "506453e5-4dc2-4acb-a624-4d0cad7bd388" (UID: "506453e5-4dc2-4acb-a624-4d0cad7bd388"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:33:13.819443 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.819395 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "506453e5-4dc2-4acb-a624-4d0cad7bd388" (UID: "506453e5-4dc2-4acb-a624-4d0cad7bd388"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:33:13.819884 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.819857 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/506453e5-4dc2-4acb-a624-4d0cad7bd388-kube-api-access-dtmv6" (OuterVolumeSpecName: "kube-api-access-dtmv6") pod "506453e5-4dc2-4acb-a624-4d0cad7bd388" (UID: "506453e5-4dc2-4acb-a624-4d0cad7bd388"). InnerVolumeSpecName "kube-api-access-dtmv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:33:13.820975 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.820892 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506453e5-4dc2-4acb-a624-4d0cad7bd388-config-out" (OuterVolumeSpecName: "config-out") pod "506453e5-4dc2-4acb-a624-4d0cad7bd388" (UID: "506453e5-4dc2-4acb-a624-4d0cad7bd388"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:33:13.820975 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.820935 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "506453e5-4dc2-4acb-a624-4d0cad7bd388" (UID: "506453e5-4dc2-4acb-a624-4d0cad7bd388"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:33:13.820975 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.820959 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "506453e5-4dc2-4acb-a624-4d0cad7bd388" (UID: "506453e5-4dc2-4acb-a624-4d0cad7bd388"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:33:13.829501 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.829480 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-web-config" (OuterVolumeSpecName: "web-config") pod "506453e5-4dc2-4acb-a624-4d0cad7bd388" (UID: "506453e5-4dc2-4acb-a624-4d0cad7bd388"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:33:13.914800 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.914767 2577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:13.914800 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.914796 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:13.914800 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.914807 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:13.914800 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.914833 2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/506453e5-4dc2-4acb-a624-4d0cad7bd388-config-out\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:13.915071 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.914841 2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/506453e5-4dc2-4acb-a624-4d0cad7bd388-tls-assets\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:13.915071 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.914850 2577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-metrics-client-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:13.915071 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.914860 2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:13.915071 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.914868 2577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:13.915071 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.914877 2577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:13.915071 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.914885 2577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-config\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:13.915071 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.914893 2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:13.915071 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.914902 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:13.915071 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.914910 2577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-kube-rbac-proxy\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:13.915071 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.914918 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtmv6\" (UniqueName: \"kubernetes.io/projected/506453e5-4dc2-4acb-a624-4d0cad7bd388-kube-api-access-dtmv6\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:13.915071 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.914926 2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:13.915071 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.914936 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-db\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:13.915071 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.914944 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/506453e5-4dc2-4acb-a624-4d0cad7bd388-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:13.915071 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.914952 2577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-secret-grpc-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:13.915071 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:13.914960 2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/506453e5-4dc2-4acb-a624-4d0cad7bd388-web-config\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:33:14.357797 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.357766 2577 generic.go:356] "Generic (PLEG): container finished" podID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerID="3e58ac03cc0ad1de0fbb5ca2e99a613b0c2d83bf9b50d72b69b79cfca1d6b696" exitCode=0 Sep 09 02:33:14.358213 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.357852 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"506453e5-4dc2-4acb-a624-4d0cad7bd388","Type":"ContainerDied","Data":"3e58ac03cc0ad1de0fbb5ca2e99a613b0c2d83bf9b50d72b69b79cfca1d6b696"} Sep 09 02:33:14.358213 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.357897 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.358213 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.357934 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"506453e5-4dc2-4acb-a624-4d0cad7bd388","Type":"ContainerDied","Data":"ac707c897c0e903228ae78e5b9d0dfcba53fcb93d595d4a056e2e375bcbc029e"} Sep 09 02:33:14.358213 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.357971 2577 scope.go:117] "RemoveContainer" containerID="f88a0e8931d28b358fe079835a5ff2d83f4e78462d83b8d5f2629897d6becda4" Sep 09 02:33:14.382623 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.382594 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 02:33:14.395261 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.395228 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 02:33:14.425532 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.425504 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 02:33:14.425870 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.425853 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="config-reloader" Sep 09 02:33:14.425870 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.425870 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="config-reloader" Sep 09 02:33:14.425965 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.425884 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="kube-rbac-proxy-thanos" Sep 09 02:33:14.425965 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.425893 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="kube-rbac-proxy-thanos" Sep 09 02:33:14.425965 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.425920 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="init-config-reloader" Sep 09 02:33:14.425965 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.425929 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="init-config-reloader" Sep 09 02:33:14.425965 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.425946 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="kube-rbac-proxy-web" Sep 09 02:33:14.425965 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.425954 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="kube-rbac-proxy-web" Sep 09 02:33:14.425965 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.425965 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="prometheus" Sep 09 02:33:14.426215 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.425972 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="prometheus" Sep 09 02:33:14.426215 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.425987 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="thanos-sidecar" Sep 09 02:33:14.426215 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.425992 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="thanos-sidecar" Sep 09 02:33:14.426215 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.425997 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="kube-rbac-proxy" Sep 09 02:33:14.426215 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.426003 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="kube-rbac-proxy" Sep 09 02:33:14.426215 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.426071 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="kube-rbac-proxy" Sep 09 02:33:14.426215 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.426086 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="kube-rbac-proxy-web" Sep 09 02:33:14.426215 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.426095 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="prometheus" Sep 09 02:33:14.426215 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.426103 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="thanos-sidecar" Sep 09 02:33:14.426215 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.426112 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="config-reloader" Sep 09 02:33:14.426215 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.426118 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" containerName="kube-rbac-proxy-thanos" Sep 09 02:33:14.431087 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.431068 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.435582 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.435563 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 09 02:33:14.435705 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.435566 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 09 02:33:14.436974 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.436940 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 09 02:33:14.437079 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.437011 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 09 02:33:14.437305 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.437271 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 09 02:33:14.437411 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.437327 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 09 02:33:14.437997 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.437972 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tbbb2\"" Sep 09 02:33:14.437997 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.437987 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 09 02:33:14.438154 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.437975 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 09 02:33:14.438212 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.438190 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 09 02:33:14.438270 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.438229 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 09 02:33:14.438395 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.438373 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1bvfn8k5em701\"" Sep 09 02:33:14.439482 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.439282 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 09 02:33:14.439651 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.439614 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 09 02:33:14.439745 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.439680 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 09 02:33:14.440431 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.440386 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 09 02:33:14.443081 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.443066 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 09 02:33:14.450518 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.450461 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 02:33:14.520652 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.520619 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73dcafda-97f5-4955-9bb2-cdfbf232eae4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.520652 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.520653 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73dcafda-97f5-4955-9bb2-cdfbf232eae4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.520903 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.520686 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/73dcafda-97f5-4955-9bb2-cdfbf232eae4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.520903 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.520764 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/73dcafda-97f5-4955-9bb2-cdfbf232eae4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.520903 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.520806 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73dcafda-97f5-4955-9bb2-cdfbf232eae4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.520903 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.520884 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.521078 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.520910 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/73dcafda-97f5-4955-9bb2-cdfbf232eae4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.521078 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.520929 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-config\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.521078 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.520945 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.521078 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.520959 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.521078 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.520980 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67nfg\" (UniqueName: \"kubernetes.io/projected/73dcafda-97f5-4955-9bb2-cdfbf232eae4-kube-api-access-67nfg\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.521078 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.521071 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.521312 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.521104 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-web-config\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.521312 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.521132 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.521312 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.521176 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73dcafda-97f5-4955-9bb2-cdfbf232eae4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.521312 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.521240 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/73dcafda-97f5-4955-9bb2-cdfbf232eae4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.521312 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.521300 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.521490 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.521322 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/73dcafda-97f5-4955-9bb2-cdfbf232eae4-config-out\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.521490 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.521346 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/73dcafda-97f5-4955-9bb2-cdfbf232eae4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.521490 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.521368 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.622246 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.622205 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.622427 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.622268 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/73dcafda-97f5-4955-9bb2-cdfbf232eae4-config-out\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.622427 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.622291 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/73dcafda-97f5-4955-9bb2-cdfbf232eae4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.622427 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.622315 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.622427 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.622341 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73dcafda-97f5-4955-9bb2-cdfbf232eae4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.622427 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.622362 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73dcafda-97f5-4955-9bb2-cdfbf232eae4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.622427 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.622408 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/73dcafda-97f5-4955-9bb2-cdfbf232eae4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.622746 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.622440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/73dcafda-97f5-4955-9bb2-cdfbf232eae4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.622746 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.622465 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73dcafda-97f5-4955-9bb2-cdfbf232eae4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.622746 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.622501 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.622746 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.622525 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/73dcafda-97f5-4955-9bb2-cdfbf232eae4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.622746 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.622550 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-config\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.622746 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.622572 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.622746 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.622595 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.622746 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.622621 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67nfg\" (UniqueName: \"kubernetes.io/projected/73dcafda-97f5-4955-9bb2-cdfbf232eae4-kube-api-access-67nfg\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.623162 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.623017 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/73dcafda-97f5-4955-9bb2-cdfbf232eae4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.623220 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.623201 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/73dcafda-97f5-4955-9bb2-cdfbf232eae4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.623274 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.623220 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73dcafda-97f5-4955-9bb2-cdfbf232eae4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.623330 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.623300 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/73dcafda-97f5-4955-9bb2-cdfbf232eae4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.623388 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.623340 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.623388 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.623370 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-web-config\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.623482 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.623396 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.623482 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.623423 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73dcafda-97f5-4955-9bb2-cdfbf232eae4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.623482 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.623450 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/73dcafda-97f5-4955-9bb2-cdfbf232eae4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.623686 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.623647 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73dcafda-97f5-4955-9bb2-cdfbf232eae4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.625666 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.625636 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.625829 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.625790 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.628388 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.628131 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-config\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.628388 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.628253 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.628547 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.628461 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/73dcafda-97f5-4955-9bb2-cdfbf232eae4-config-out\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.628605 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.628564 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-web-config\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.629168 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.625885 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73dcafda-97f5-4955-9bb2-cdfbf232eae4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.629168 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.629066 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.629331 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.629296 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/73dcafda-97f5-4955-9bb2-cdfbf232eae4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.630190 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.629425 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.630190 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.629469 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.631880 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.631153 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/73dcafda-97f5-4955-9bb2-cdfbf232eae4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.633027 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.633001 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73dcafda-97f5-4955-9bb2-cdfbf232eae4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.635082 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.635056 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/73dcafda-97f5-4955-9bb2-cdfbf232eae4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.636041 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.636021 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67nfg\" (UniqueName: \"kubernetes.io/projected/73dcafda-97f5-4955-9bb2-cdfbf232eae4-kube-api-access-67nfg\") pod \"prometheus-k8s-0\" (UID: \"73dcafda-97f5-4955-9bb2-cdfbf232eae4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.742234 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.742143 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:14.778792 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.778774 2577 scope.go:117] "RemoveContainer" containerID="ce2fbe1e33ef3624ce696a0de2ce445893e32375d5a3a95f65115ff80a1cdb30" Sep 09 02:33:14.823307 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.823278 2577 scope.go:117] "RemoveContainer" containerID="3e58ac03cc0ad1de0fbb5ca2e99a613b0c2d83bf9b50d72b69b79cfca1d6b696" Sep 09 02:33:14.837933 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.837908 2577 scope.go:117] "RemoveContainer" containerID="474b5638ef5b5010c100d4a19060fbeb8d48b08ff266252ece3eec12a1ad415b" Sep 09 02:33:14.845491 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.845466 2577 scope.go:117] "RemoveContainer" containerID="239fae9ef6559236f9104bc289129207a19ebfe6afdca958c2335f8c87e167d8" Sep 09 02:33:14.854707 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.854679 2577 scope.go:117] "RemoveContainer" containerID="d8e874c5c44b710e695e2cea211c5e37baaa80d79e430db1b5219467b624e449" Sep 09 02:33:14.862356 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.862332 2577 scope.go:117] "RemoveContainer" containerID="c1c5fb084a525b506e529842f2a71dd23a7e200997b81e8221e36610c6495d0e" Sep 09 02:33:14.876381 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.876364 2577 scope.go:117] "RemoveContainer" containerID="f88a0e8931d28b358fe079835a5ff2d83f4e78462d83b8d5f2629897d6becda4" Sep 09 02:33:14.876669 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:33:14.876647 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f88a0e8931d28b358fe079835a5ff2d83f4e78462d83b8d5f2629897d6becda4\": container with ID starting with f88a0e8931d28b358fe079835a5ff2d83f4e78462d83b8d5f2629897d6becda4 not found: ID does not exist" containerID="f88a0e8931d28b358fe079835a5ff2d83f4e78462d83b8d5f2629897d6becda4" Sep 09 02:33:14.876742 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.876683 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f88a0e8931d28b358fe079835a5ff2d83f4e78462d83b8d5f2629897d6becda4"} err="failed to get container status \"f88a0e8931d28b358fe079835a5ff2d83f4e78462d83b8d5f2629897d6becda4\": rpc error: code = NotFound desc = could not find container \"f88a0e8931d28b358fe079835a5ff2d83f4e78462d83b8d5f2629897d6becda4\": container with ID starting with f88a0e8931d28b358fe079835a5ff2d83f4e78462d83b8d5f2629897d6becda4 not found: ID does not exist" Sep 09 02:33:14.876742 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.876708 2577 scope.go:117] "RemoveContainer" containerID="ce2fbe1e33ef3624ce696a0de2ce445893e32375d5a3a95f65115ff80a1cdb30" Sep 09 02:33:14.877007 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:33:14.876991 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2fbe1e33ef3624ce696a0de2ce445893e32375d5a3a95f65115ff80a1cdb30\": container with ID starting with ce2fbe1e33ef3624ce696a0de2ce445893e32375d5a3a95f65115ff80a1cdb30 not found: ID does not exist" containerID="ce2fbe1e33ef3624ce696a0de2ce445893e32375d5a3a95f65115ff80a1cdb30" Sep 09 02:33:14.877080 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.877015 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2fbe1e33ef3624ce696a0de2ce445893e32375d5a3a95f65115ff80a1cdb30"} err="failed to get container status \"ce2fbe1e33ef3624ce696a0de2ce445893e32375d5a3a95f65115ff80a1cdb30\": rpc error: code = NotFound desc = could not find container \"ce2fbe1e33ef3624ce696a0de2ce445893e32375d5a3a95f65115ff80a1cdb30\": container with ID starting with ce2fbe1e33ef3624ce696a0de2ce445893e32375d5a3a95f65115ff80a1cdb30 not found: ID does not exist" Sep 09 02:33:14.877080 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.877035 2577 scope.go:117] "RemoveContainer" containerID="3e58ac03cc0ad1de0fbb5ca2e99a613b0c2d83bf9b50d72b69b79cfca1d6b696" Sep 09 02:33:14.877291 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:33:14.877276 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e58ac03cc0ad1de0fbb5ca2e99a613b0c2d83bf9b50d72b69b79cfca1d6b696\": container with ID starting with 3e58ac03cc0ad1de0fbb5ca2e99a613b0c2d83bf9b50d72b69b79cfca1d6b696 not found: ID does not exist" containerID="3e58ac03cc0ad1de0fbb5ca2e99a613b0c2d83bf9b50d72b69b79cfca1d6b696" Sep 09 02:33:14.877353 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.877298 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e58ac03cc0ad1de0fbb5ca2e99a613b0c2d83bf9b50d72b69b79cfca1d6b696"} err="failed to get container status \"3e58ac03cc0ad1de0fbb5ca2e99a613b0c2d83bf9b50d72b69b79cfca1d6b696\": rpc error: code = NotFound desc = could not find container \"3e58ac03cc0ad1de0fbb5ca2e99a613b0c2d83bf9b50d72b69b79cfca1d6b696\": container with ID starting with 3e58ac03cc0ad1de0fbb5ca2e99a613b0c2d83bf9b50d72b69b79cfca1d6b696 not found: ID does not exist" Sep 09 02:33:14.877353 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.877317 2577 scope.go:117] "RemoveContainer" containerID="474b5638ef5b5010c100d4a19060fbeb8d48b08ff266252ece3eec12a1ad415b" Sep 09 02:33:14.877596 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:33:14.877576 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"474b5638ef5b5010c100d4a19060fbeb8d48b08ff266252ece3eec12a1ad415b\": container with ID starting with 474b5638ef5b5010c100d4a19060fbeb8d48b08ff266252ece3eec12a1ad415b not found: ID does not exist" containerID="474b5638ef5b5010c100d4a19060fbeb8d48b08ff266252ece3eec12a1ad415b" Sep 09 02:33:14.877649 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.877601 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474b5638ef5b5010c100d4a19060fbeb8d48b08ff266252ece3eec12a1ad415b"} err="failed to get container status \"474b5638ef5b5010c100d4a19060fbeb8d48b08ff266252ece3eec12a1ad415b\": rpc error: code = NotFound desc = could not find container \"474b5638ef5b5010c100d4a19060fbeb8d48b08ff266252ece3eec12a1ad415b\": container with ID starting with 474b5638ef5b5010c100d4a19060fbeb8d48b08ff266252ece3eec12a1ad415b not found: ID does not exist" Sep 09 02:33:14.877649 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.877618 2577 scope.go:117] "RemoveContainer" containerID="239fae9ef6559236f9104bc289129207a19ebfe6afdca958c2335f8c87e167d8" Sep 09 02:33:14.877877 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:33:14.877859 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"239fae9ef6559236f9104bc289129207a19ebfe6afdca958c2335f8c87e167d8\": container with ID starting with 239fae9ef6559236f9104bc289129207a19ebfe6afdca958c2335f8c87e167d8 not found: ID does not exist" containerID="239fae9ef6559236f9104bc289129207a19ebfe6afdca958c2335f8c87e167d8" Sep 09 02:33:14.877945 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.877884 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239fae9ef6559236f9104bc289129207a19ebfe6afdca958c2335f8c87e167d8"} err="failed to get container status \"239fae9ef6559236f9104bc289129207a19ebfe6afdca958c2335f8c87e167d8\": rpc error: code = NotFound desc = could not find container \"239fae9ef6559236f9104bc289129207a19ebfe6afdca958c2335f8c87e167d8\": container with ID starting with 239fae9ef6559236f9104bc289129207a19ebfe6afdca958c2335f8c87e167d8 not found: ID does not exist" Sep 09 02:33:14.877945 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.877902 2577 scope.go:117] "RemoveContainer" containerID="d8e874c5c44b710e695e2cea211c5e37baaa80d79e430db1b5219467b624e449" Sep 09 02:33:14.878166 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:33:14.878151 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8e874c5c44b710e695e2cea211c5e37baaa80d79e430db1b5219467b624e449\": container with ID starting with d8e874c5c44b710e695e2cea211c5e37baaa80d79e430db1b5219467b624e449 not found: ID does not exist" containerID="d8e874c5c44b710e695e2cea211c5e37baaa80d79e430db1b5219467b624e449" Sep 09 02:33:14.878223 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.878171 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8e874c5c44b710e695e2cea211c5e37baaa80d79e430db1b5219467b624e449"} err="failed to get container status \"d8e874c5c44b710e695e2cea211c5e37baaa80d79e430db1b5219467b624e449\": rpc error: code = NotFound desc = could not find container \"d8e874c5c44b710e695e2cea211c5e37baaa80d79e430db1b5219467b624e449\": container with ID starting with d8e874c5c44b710e695e2cea211c5e37baaa80d79e430db1b5219467b624e449 not found: ID does not exist" Sep 09 02:33:14.878223 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.878188 2577 scope.go:117] "RemoveContainer" containerID="c1c5fb084a525b506e529842f2a71dd23a7e200997b81e8221e36610c6495d0e" Sep 09 02:33:14.878418 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:33:14.878400 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c5fb084a525b506e529842f2a71dd23a7e200997b81e8221e36610c6495d0e\": container with ID starting with c1c5fb084a525b506e529842f2a71dd23a7e200997b81e8221e36610c6495d0e not found: ID does not exist" containerID="c1c5fb084a525b506e529842f2a71dd23a7e200997b81e8221e36610c6495d0e" Sep 09 02:33:14.878465 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:14.878422 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c5fb084a525b506e529842f2a71dd23a7e200997b81e8221e36610c6495d0e"} err="failed to get container status \"c1c5fb084a525b506e529842f2a71dd23a7e200997b81e8221e36610c6495d0e\": rpc error: code = NotFound desc = could not find container \"c1c5fb084a525b506e529842f2a71dd23a7e200997b81e8221e36610c6495d0e\": container with ID starting with c1c5fb084a525b506e529842f2a71dd23a7e200997b81e8221e36610c6495d0e not found: ID does not exist" Sep 09 02:33:15.066261 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:15.066224 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 02:33:15.068691 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:33:15.068667 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73dcafda_97f5_4955_9bb2_cdfbf232eae4.slice/crio-9ab40c31441cacdbeb92f89442cb941e3f450bf2743c3f97c06b51b3797e98df WatchSource:0}: Error finding container 9ab40c31441cacdbeb92f89442cb941e3f450bf2743c3f97c06b51b3797e98df: Status 404 returned error can't find the container with id 9ab40c31441cacdbeb92f89442cb941e3f450bf2743c3f97c06b51b3797e98df Sep 09 02:33:15.366306 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:15.366183 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" event={"ID":"ea9506c8-63c6-4868-b855-d938d9858e35","Type":"ContainerStarted","Data":"aa1f175816f8e2e18b440c57b037a770a8d3333823b5729fc010dee705474302"} Sep 09 02:33:15.366306 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:15.366248 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" event={"ID":"ea9506c8-63c6-4868-b855-d938d9858e35","Type":"ContainerStarted","Data":"18f96ec1c56b66bab5aac8c4a5991071b70752d61ce4ff8ad9b272293bc9d1f9"} Sep 09 02:33:15.366306 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:15.366277 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" event={"ID":"ea9506c8-63c6-4868-b855-d938d9858e35","Type":"ContainerStarted","Data":"ece6512080a5ecdb9f8c6fc1cd2d4d6056355b20b9c02cb08cafd53fdf72bff5"} Sep 09 02:33:15.367878 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:15.367854 2577 generic.go:356] "Generic (PLEG): container finished" podID="73dcafda-97f5-4955-9bb2-cdfbf232eae4" containerID="00407c775ad0fcdc600586348888e81552c8449433b40368702d23783a90929f" exitCode=0 Sep 09 02:33:15.368017 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:15.367887 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73dcafda-97f5-4955-9bb2-cdfbf232eae4","Type":"ContainerDied","Data":"00407c775ad0fcdc600586348888e81552c8449433b40368702d23783a90929f"} Sep 09 02:33:15.368017 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:15.367941 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73dcafda-97f5-4955-9bb2-cdfbf232eae4","Type":"ContainerStarted","Data":"9ab40c31441cacdbeb92f89442cb941e3f450bf2743c3f97c06b51b3797e98df"} Sep 09 02:33:15.396667 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:15.396623 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-54b8dcbdd-9zrn6" podStartSLOduration=1.347647684 podStartE2EDuration="3.396606209s" totalImagesPullingTime="2.048958525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:33:15.395803777 +0000 UTC m=+183.526045873" watchObservedRunningTime="2026-09-09 02:33:15.396606209 +0000 UTC m=+183.526848306" Sep 09 02:33:15.673490 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:15.673464 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:33:15.775966 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:15.775940 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:33:16.309669 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:16.309635 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="506453e5-4dc2-4acb-a624-4d0cad7bd388" path="/var/lib/kubelet/pods/506453e5-4dc2-4acb-a624-4d0cad7bd388/volumes" Sep 09 02:33:16.375732 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:16.375691 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73dcafda-97f5-4955-9bb2-cdfbf232eae4","Type":"ContainerStarted","Data":"dfaa67645df4cd837ce81afd44e2097bf9e93d6c2f4ed3d9910602ee5c4e2d25"} Sep 09 02:33:16.376171 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:16.375756 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73dcafda-97f5-4955-9bb2-cdfbf232eae4","Type":"ContainerStarted","Data":"36c34204ca94d28e7a876cec157ecf3e08232fb829fa9bad7275e4bde4f938bd"} Sep 09 02:33:16.376171 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:16.375785 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73dcafda-97f5-4955-9bb2-cdfbf232eae4","Type":"ContainerStarted","Data":"1f949c2f32450e74491e6d6c5a7f0b6f011309186cd6f91c2cfc4c3c7cd2bddf"} Sep 09 02:33:16.376171 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:16.375826 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73dcafda-97f5-4955-9bb2-cdfbf232eae4","Type":"ContainerStarted","Data":"307ddf11a6c35a3e6e241ae614dade33fa0c60064fe973800eeed0ba79c4cfc1"} Sep 09 02:33:16.376171 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:16.375862 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73dcafda-97f5-4955-9bb2-cdfbf232eae4","Type":"ContainerStarted","Data":"e9fdfbf48aee46a2af91605adcb63b48d5e155a594af216628ee52ca7a262ec8"} Sep 09 02:33:16.376171 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:16.375888 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73dcafda-97f5-4955-9bb2-cdfbf232eae4","Type":"ContainerStarted","Data":"230bd02731c7e7a763ad768b7b75075fc9446779321c2141e34cf455e7cf6598"} Sep 09 02:33:16.409973 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:16.409850 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.389779933 podStartE2EDuration="2.409833436s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.053503ms" isStatefulPod=false podCreationTimestamp="2026-09-09 02:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:33:16.408187922 +0000 UTC m=+184.538430018" watchObservedRunningTime="2026-09-09 02:33:16.409833436 +0000 UTC m=+184.540075523" Sep 09 02:33:19.743269 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:19.743227 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:33:26.194333 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:26.194298 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:33:26.294577 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:26.294546 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:33:26.372517 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:26.372483 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:33:56.615179 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:33:56.615144 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:34:09.580059 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:34:09.580024 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:34:09.651508 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:34:09.651475 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:34:14.742665 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:34:14.742628 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:34:14.758101 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:34:14.758069 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:34:15.633457 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:34:15.633424 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 02:34:22.407632 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:34:22.407599 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:34:22.500033 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:34:22.499999 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:34:25.172699 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:34:25.172670 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:34:34.334989 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:34:34.334945 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:34:34.435665 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:34:34.435632 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:34:43.190095 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:34:43.190017 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:34:43.290749 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:34:43.290713 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:35:03.349299 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:35:03.349266 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:35:03.446086 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:35:03.446045 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:35:03.547573 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:35:03.547547 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:35:04.137845 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:35:04.134060 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:35:04.230151 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:35:04.230118 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:35:12.251343 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:35:12.251312 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/2.log" Sep 09 02:35:12.254522 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:35:12.254493 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/2.log" Sep 09 02:35:12.261639 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:35:12.261624 2577 kubelet.go:1648] "Image garbage collection succeeded" Sep 09 02:35:14.639812 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:35:14.639767 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:35:14.738064 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:35:14.738021 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:35:35.232442 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:35:35.232406 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:35:36.420631 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:35:36.420598 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:35:53.579236 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:35:53.579201 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:35:53.683284 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:35:53.683254 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:36:25.885043 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:25.884956 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:36:26.665358 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:26.665328 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:36:29.702894 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:29.702859 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:36:29.808299 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:29.808268 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:36:31.211328 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:31.211292 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:36:31.377707 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:31.377676 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:36:31.547682 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:31.547594 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:36:33.441332 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:33.441297 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:36:33.835293 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:33.835196 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:36:34.521502 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:34.521468 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-szk5f"] Sep 09 02:36:34.525018 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:34.524998 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-szk5f" Sep 09 02:36:34.527920 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:34.527901 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 09 02:36:34.528027 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:34.527937 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 09 02:36:34.529229 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:34.529214 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-w5tw4\"" Sep 09 02:36:34.536508 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:34.536486 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-szk5f"] Sep 09 02:36:34.659540 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:34.659507 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2724c080-cd29-4500-a353-3662c1e4f116-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-szk5f\" (UID: \"2724c080-cd29-4500-a353-3662c1e4f116\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-szk5f" Sep 09 02:36:34.659702 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:34.659552 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57z2v\" (UniqueName: \"kubernetes.io/projected/2724c080-cd29-4500-a353-3662c1e4f116-kube-api-access-57z2v\") pod \"cert-manager-cainjector-559cdd8c87-szk5f\" (UID: \"2724c080-cd29-4500-a353-3662c1e4f116\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-szk5f" Sep 09 02:36:34.759979 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:34.759951 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2724c080-cd29-4500-a353-3662c1e4f116-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-szk5f\" (UID: \"2724c080-cd29-4500-a353-3662c1e4f116\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-szk5f" Sep 09 02:36:34.760160 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:34.760016 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57z2v\" (UniqueName: \"kubernetes.io/projected/2724c080-cd29-4500-a353-3662c1e4f116-kube-api-access-57z2v\") pod \"cert-manager-cainjector-559cdd8c87-szk5f\" (UID: \"2724c080-cd29-4500-a353-3662c1e4f116\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-szk5f" Sep 09 02:36:34.770725 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:34.770691 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57z2v\" (UniqueName: \"kubernetes.io/projected/2724c080-cd29-4500-a353-3662c1e4f116-kube-api-access-57z2v\") pod \"cert-manager-cainjector-559cdd8c87-szk5f\" (UID: \"2724c080-cd29-4500-a353-3662c1e4f116\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-szk5f" Sep 09 02:36:34.771186 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:34.771167 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2724c080-cd29-4500-a353-3662c1e4f116-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-szk5f\" (UID: \"2724c080-cd29-4500-a353-3662c1e4f116\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-szk5f" Sep 09 02:36:34.846166 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:34.846098 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-szk5f" Sep 09 02:36:35.118917 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:35.118890 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-szk5f"] Sep 09 02:36:35.119878 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:36:35.119851 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2724c080_cd29_4500_a353_3662c1e4f116.slice/crio-6e51a40ccc9d48499b70e323325dd195cf22a2cc7de29c163a6d335694384576 WatchSource:0}: Error finding container 6e51a40ccc9d48499b70e323325dd195cf22a2cc7de29c163a6d335694384576: Status 404 returned error can't find the container with id 6e51a40ccc9d48499b70e323325dd195cf22a2cc7de29c163a6d335694384576 Sep 09 02:36:35.121763 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:35.121746 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 02:36:35.187096 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:35.187060 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-szk5f" event={"ID":"2724c080-cd29-4500-a353-3662c1e4f116","Type":"ContainerStarted","Data":"6e51a40ccc9d48499b70e323325dd195cf22a2cc7de29c163a6d335694384576"} Sep 09 02:36:36.689669 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:36.689635 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:36:36.794158 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:36.794124 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:36:40.211492 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:40.211458 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-szk5f" event={"ID":"2724c080-cd29-4500-a353-3662c1e4f116","Type":"ContainerStarted","Data":"cacb68a603046854a90901384f16ba5967b601f4b187b1edc98f387131fdac87"} Sep 09 02:36:40.235076 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:40.235027 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-szk5f" podStartSLOduration=1.866629975 podStartE2EDuration="6.235011988s" totalImagesPullingTime="4.368382013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:36:40.233307657 +0000 UTC m=+388.363549752" watchObservedRunningTime="2026-09-09 02:36:40.235011988 +0000 UTC m=+388.365254084" Sep 09 02:36:40.329650 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:40.329614 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:36:40.430693 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:40.430644 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:36:41.842144 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:41.842100 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:36:49.106457 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:49.106420 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:36:49.205371 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:49.205340 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:36:50.728546 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:50.728505 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ct2gb"] Sep 09 02:36:50.732199 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:50.732177 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ct2gb" Sep 09 02:36:50.735927 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:50.735904 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 09 02:36:50.735927 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:50.735923 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 09 02:36:50.737373 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:50.737350 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-pckm4\"" Sep 09 02:36:50.745723 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:50.745700 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ct2gb"] Sep 09 02:36:50.797779 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:50.797747 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4cec45e-ad48-4159-8ce3-5078d583e956-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ct2gb\" (UID: \"f4cec45e-ad48-4159-8ce3-5078d583e956\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ct2gb" Sep 09 02:36:50.797779 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:50.797781 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcc9q\" (UniqueName: \"kubernetes.io/projected/f4cec45e-ad48-4159-8ce3-5078d583e956-kube-api-access-xcc9q\") pod \"openshift-lws-operator-fd8ccff4c-ct2gb\" (UID: \"f4cec45e-ad48-4159-8ce3-5078d583e956\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ct2gb" Sep 09 02:36:50.899118 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:50.899087 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4cec45e-ad48-4159-8ce3-5078d583e956-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ct2gb\" (UID: \"f4cec45e-ad48-4159-8ce3-5078d583e956\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ct2gb" Sep 09 02:36:50.899358 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:50.899131 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcc9q\" (UniqueName: \"kubernetes.io/projected/f4cec45e-ad48-4159-8ce3-5078d583e956-kube-api-access-xcc9q\") pod \"openshift-lws-operator-fd8ccff4c-ct2gb\" (UID: \"f4cec45e-ad48-4159-8ce3-5078d583e956\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ct2gb" Sep 09 02:36:50.899589 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:50.899558 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4cec45e-ad48-4159-8ce3-5078d583e956-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ct2gb\" (UID: \"f4cec45e-ad48-4159-8ce3-5078d583e956\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ct2gb" Sep 09 02:36:50.915963 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:50.915930 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcc9q\" (UniqueName: \"kubernetes.io/projected/f4cec45e-ad48-4159-8ce3-5078d583e956-kube-api-access-xcc9q\") pod \"openshift-lws-operator-fd8ccff4c-ct2gb\" (UID: \"f4cec45e-ad48-4159-8ce3-5078d583e956\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ct2gb" Sep 09 02:36:51.040908 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:51.040803 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ct2gb" Sep 09 02:36:51.316635 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:51.316607 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ct2gb"] Sep 09 02:36:51.319300 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:36:51.319270 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4cec45e_ad48_4159_8ce3_5078d583e956.slice/crio-db942efe5a51a7496919655d6534bff00a972492dcf0196943b649ceb7e28d60 WatchSource:0}: Error finding container db942efe5a51a7496919655d6534bff00a972492dcf0196943b649ceb7e28d60: Status 404 returned error can't find the container with id db942efe5a51a7496919655d6534bff00a972492dcf0196943b649ceb7e28d60 Sep 09 02:36:52.263716 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:52.263671 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ct2gb" event={"ID":"f4cec45e-ad48-4159-8ce3-5078d583e956","Type":"ContainerStarted","Data":"db942efe5a51a7496919655d6534bff00a972492dcf0196943b649ceb7e28d60"} Sep 09 02:36:54.275508 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:54.275469 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ct2gb" event={"ID":"f4cec45e-ad48-4159-8ce3-5078d583e956","Type":"ContainerStarted","Data":"eb7bae4da6060e2759cda7d38d334e0d2cf6514381a8c130faea0c130ad0ad10"} Sep 09 02:36:54.306139 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:54.306082 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ct2gb" podStartSLOduration=1.914715637 podStartE2EDuration="4.306062479s" totalImagesPullingTime="2.391346842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:36:54.303953084 +0000 UTC m=+402.434195181" watchObservedRunningTime="2026-09-09 02:36:54.306062479 +0000 UTC m=+402.436304576" Sep 09 02:36:55.078890 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:55.078850 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:36:55.184033 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:55.183999 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:36:56.922724 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:36:56.922687 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:09.233994 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.233957 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:09.301940 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.301912 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c4bb7846b-q5n4w"] Sep 09 02:37:09.307089 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.307064 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-q5n4w" Sep 09 02:37:09.310479 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.310455 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 09 02:37:09.311531 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.311509 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 09 02:37:09.311531 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.311526 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-vxnzz\"" Sep 09 02:37:09.312444 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.312426 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 09 02:37:09.319685 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.319660 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c4bb7846b-q5n4w"] Sep 09 02:37:09.350472 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.350447 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:09.360683 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.360656 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6m45\" (UniqueName: \"kubernetes.io/projected/a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd-kube-api-access-j6m45\") pod \"lws-controller-manager-6c4bb7846b-q5n4w\" (UID: \"a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-q5n4w" Sep 09 02:37:09.360779 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.360706 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd-metrics-cert\") pod \"lws-controller-manager-6c4bb7846b-q5n4w\" (UID: \"a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-q5n4w" Sep 09 02:37:09.360779 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.360731 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd-cert\") pod \"lws-controller-manager-6c4bb7846b-q5n4w\" (UID: \"a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-q5n4w" Sep 09 02:37:09.360779 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.360751 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd-manager-config\") pod \"lws-controller-manager-6c4bb7846b-q5n4w\" (UID: \"a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-q5n4w" Sep 09 02:37:09.461138 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.461107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd-manager-config\") pod \"lws-controller-manager-6c4bb7846b-q5n4w\" (UID: \"a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-q5n4w" Sep 09 02:37:09.461326 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.461170 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6m45\" (UniqueName: \"kubernetes.io/projected/a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd-kube-api-access-j6m45\") pod \"lws-controller-manager-6c4bb7846b-q5n4w\" (UID: \"a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-q5n4w" Sep 09 02:37:09.461326 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.461230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd-metrics-cert\") pod \"lws-controller-manager-6c4bb7846b-q5n4w\" (UID: \"a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-q5n4w" Sep 09 02:37:09.461326 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.461268 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd-cert\") pod \"lws-controller-manager-6c4bb7846b-q5n4w\" (UID: \"a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-q5n4w" Sep 09 02:37:09.461910 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.461885 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd-manager-config\") pod \"lws-controller-manager-6c4bb7846b-q5n4w\" (UID: \"a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-q5n4w" Sep 09 02:37:09.463773 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.463746 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd-metrics-cert\") pod \"lws-controller-manager-6c4bb7846b-q5n4w\" (UID: \"a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-q5n4w" Sep 09 02:37:09.464023 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.464004 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd-cert\") pod \"lws-controller-manager-6c4bb7846b-q5n4w\" (UID: \"a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-q5n4w" Sep 09 02:37:09.473523 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.473498 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6m45\" (UniqueName: \"kubernetes.io/projected/a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd-kube-api-access-j6m45\") pod \"lws-controller-manager-6c4bb7846b-q5n4w\" (UID: \"a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-q5n4w" Sep 09 02:37:09.617004 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.616919 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-q5n4w" Sep 09 02:37:09.731316 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.731288 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:09.866139 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.866105 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:09.911547 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:09.911516 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c4bb7846b-q5n4w"] Sep 09 02:37:09.914786 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:37:09.914761 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e31d9f_25f3_46c1_a37e_ddec8e1c92fd.slice/crio-90f6ef43fa01b941ea8a3edae80a51acc419536a4713f7b75bec2af642eb71c9 WatchSource:0}: Error finding container 90f6ef43fa01b941ea8a3edae80a51acc419536a4713f7b75bec2af642eb71c9: Status 404 returned error can't find the container with id 90f6ef43fa01b941ea8a3edae80a51acc419536a4713f7b75bec2af642eb71c9 Sep 09 02:37:10.341701 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:10.341618 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-q5n4w" event={"ID":"a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd","Type":"ContainerStarted","Data":"90f6ef43fa01b941ea8a3edae80a51acc419536a4713f7b75bec2af642eb71c9"} Sep 09 02:37:10.374445 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:10.374417 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:10.788725 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:10.788692 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:10.902131 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:10.902096 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:10.983616 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:10.983572 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2"] Sep 09 02:37:10.986124 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:10.986098 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" Sep 09 02:37:10.991756 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:10.991737 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 09 02:37:10.992342 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:10.992061 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 09 02:37:10.992342 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:10.992070 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b6csz\"" Sep 09 02:37:10.992342 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:10.992144 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 09 02:37:10.992342 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:10.992065 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 09 02:37:11.003851 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.003811 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2"] Sep 09 02:37:11.070476 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.070397 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22p2j\" (UniqueName: \"kubernetes.io/projected/0377c506-1c88-4580-bcfc-c81bd78e3f30-kube-api-access-22p2j\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-4gjx2\" (UID: \"0377c506-1c88-4580-bcfc-c81bd78e3f30\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" Sep 09 02:37:11.070476 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.070443 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0377c506-1c88-4580-bcfc-c81bd78e3f30-tmp\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-4gjx2\" (UID: \"0377c506-1c88-4580-bcfc-c81bd78e3f30\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" Sep 09 02:37:11.070675 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.070497 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0377c506-1c88-4580-bcfc-c81bd78e3f30-manifests\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-4gjx2\" (UID: \"0377c506-1c88-4580-bcfc-c81bd78e3f30\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" Sep 09 02:37:11.070675 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.070541 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0377c506-1c88-4580-bcfc-c81bd78e3f30-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-4gjx2\" (UID: \"0377c506-1c88-4580-bcfc-c81bd78e3f30\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" Sep 09 02:37:11.070675 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.070574 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0377c506-1c88-4580-bcfc-c81bd78e3f30-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-4gjx2\" (UID: \"0377c506-1c88-4580-bcfc-c81bd78e3f30\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" Sep 09 02:37:11.171102 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.171074 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0377c506-1c88-4580-bcfc-c81bd78e3f30-manifests\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-4gjx2\" (UID: \"0377c506-1c88-4580-bcfc-c81bd78e3f30\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" Sep 09 02:37:11.171265 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.171111 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0377c506-1c88-4580-bcfc-c81bd78e3f30-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-4gjx2\" (UID: \"0377c506-1c88-4580-bcfc-c81bd78e3f30\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" Sep 09 02:37:11.171265 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.171133 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0377c506-1c88-4580-bcfc-c81bd78e3f30-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-4gjx2\" (UID: \"0377c506-1c88-4580-bcfc-c81bd78e3f30\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" Sep 09 02:37:11.171265 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.171203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22p2j\" (UniqueName: \"kubernetes.io/projected/0377c506-1c88-4580-bcfc-c81bd78e3f30-kube-api-access-22p2j\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-4gjx2\" (UID: \"0377c506-1c88-4580-bcfc-c81bd78e3f30\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" Sep 09 02:37:11.171265 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.171232 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0377c506-1c88-4580-bcfc-c81bd78e3f30-tmp\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-4gjx2\" (UID: \"0377c506-1c88-4580-bcfc-c81bd78e3f30\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" Sep 09 02:37:11.171489 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.171467 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0377c506-1c88-4580-bcfc-c81bd78e3f30-manifests\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-4gjx2\" (UID: \"0377c506-1c88-4580-bcfc-c81bd78e3f30\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" Sep 09 02:37:11.173656 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.173631 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0377c506-1c88-4580-bcfc-c81bd78e3f30-tmp\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-4gjx2\" (UID: \"0377c506-1c88-4580-bcfc-c81bd78e3f30\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" Sep 09 02:37:11.174002 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.173976 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0377c506-1c88-4580-bcfc-c81bd78e3f30-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-4gjx2\" (UID: \"0377c506-1c88-4580-bcfc-c81bd78e3f30\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" Sep 09 02:37:11.174103 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.174032 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0377c506-1c88-4580-bcfc-c81bd78e3f30-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-4gjx2\" (UID: \"0377c506-1c88-4580-bcfc-c81bd78e3f30\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" Sep 09 02:37:11.176028 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.176006 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:11.186770 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.186751 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22p2j\" (UniqueName: \"kubernetes.io/projected/0377c506-1c88-4580-bcfc-c81bd78e3f30-kube-api-access-22p2j\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-4gjx2\" (UID: \"0377c506-1c88-4580-bcfc-c81bd78e3f30\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" Sep 09 02:37:11.299089 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.299049 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" Sep 09 02:37:11.307883 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.307856 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:11.406715 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.406680 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:11.511033 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.511003 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:11.597515 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.597491 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2"] Sep 09 02:37:11.598461 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.598436 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:11.679590 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.679547 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:11.686979 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:37:11.686952 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0377c506_1c88_4580_bcfc_c81bd78e3f30.slice/crio-fea823ddfe176781a2e3f994cffe93a3be296dee58fb36040f8e79a423773f58 WatchSource:0}: Error finding container fea823ddfe176781a2e3f994cffe93a3be296dee58fb36040f8e79a423773f58: Status 404 returned error can't find the container with id fea823ddfe176781a2e3f994cffe93a3be296dee58fb36040f8e79a423773f58 Sep 09 02:37:11.788314 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:11.788288 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:12.355622 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:12.355583 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-q5n4w" event={"ID":"a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd","Type":"ContainerStarted","Data":"b2c0847e97f0904f5117397972ff829273748e1391e97c167753adb0f32f1a58"} Sep 09 02:37:12.355868 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:12.355849 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-q5n4w" Sep 09 02:37:12.357711 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:12.357676 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" event={"ID":"0377c506-1c88-4580-bcfc-c81bd78e3f30","Type":"ContainerStarted","Data":"fea823ddfe176781a2e3f994cffe93a3be296dee58fb36040f8e79a423773f58"} Sep 09 02:37:12.386880 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:12.386810 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-q5n4w" podStartSLOduration=1.583025997 podStartE2EDuration="3.386795547s" totalImagesPullingTime="1.80376955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:37:12.385361419 +0000 UTC m=+420.515603513" watchObservedRunningTime="2026-09-09 02:37:12.386795547 +0000 UTC m=+420.517037642" Sep 09 02:37:15.374452 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:15.374420 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" event={"ID":"0377c506-1c88-4580-bcfc-c81bd78e3f30","Type":"ContainerStarted","Data":"186ba53634bfb7f30a1be4b8fcabe1c85740a4cbb621b2d3e1d5f552ee8c2a50"} Sep 09 02:37:15.929705 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:15.929610 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:16.028334 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:16.028296 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:16.132212 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:16.132174 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:16.380319 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:16.380289 2577 generic.go:356] "Generic (PLEG): container finished" podID="0377c506-1c88-4580-bcfc-c81bd78e3f30" containerID="186ba53634bfb7f30a1be4b8fcabe1c85740a4cbb621b2d3e1d5f552ee8c2a50" exitCode=0 Sep 09 02:37:16.380771 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:16.380322 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" event={"ID":"0377c506-1c88-4580-bcfc-c81bd78e3f30","Type":"ContainerDied","Data":"186ba53634bfb7f30a1be4b8fcabe1c85740a4cbb621b2d3e1d5f552ee8c2a50"} Sep 09 02:37:16.625320 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:16.625285 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:16.722319 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:16.722232 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:17.387303 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:17.387269 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" event={"ID":"0377c506-1c88-4580-bcfc-c81bd78e3f30","Type":"ContainerStarted","Data":"e9cf1de032e2733042ff9b23cd05634919923ae5a370cc639c9ea7a998a8bd06"} Sep 09 02:37:17.387735 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:17.387392 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" Sep 09 02:37:17.415689 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:17.415633 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" podStartSLOduration=3.145269853 podStartE2EDuration="7.415615438s" totalImagesPullingTime="3.216508948s" totalInitContainerRuntime="1.053836637s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:37:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:37:17.412151222 +0000 UTC m=+425.542393319" watchObservedRunningTime="2026-09-09 02:37:17.415615438 +0000 UTC m=+425.545857535" Sep 09 02:37:21.466755 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:21.466724 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:21.605462 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:21.605426 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:23.365775 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:23.365747 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-q5n4w" Sep 09 02:37:26.856036 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:26.855986 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:26.954444 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:26.954413 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:27.141472 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:27.141434 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:27.243035 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:27.242994 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:28.394064 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:28.394031 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-4gjx2" Sep 09 02:37:30.534454 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:30.534414 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:30.683587 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:30.683535 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:41.437897 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:41.437798 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:41.468430 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:41.468398 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7dc95696d9-wz9vg"] Sep 09 02:37:41.471363 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:41.471348 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7dc95696d9-wz9vg" Sep 09 02:37:41.475473 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:41.475434 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 09 02:37:41.476769 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:41.476749 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-c5v6g\"" Sep 09 02:37:41.476896 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:41.476794 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 09 02:37:41.477113 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:41.477095 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 09 02:37:41.477175 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:41.477099 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 09 02:37:41.483928 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:41.483904 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7dc95696d9-wz9vg"] Sep 09 02:37:41.530162 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:41.530131 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f65ba8-8aae-4b5a-9229-0813168e1063-tls-certs\") pod \"kube-auth-proxy-7dc95696d9-wz9vg\" (UID: \"74f65ba8-8aae-4b5a-9229-0813168e1063\") " pod="openshift-ingress/kube-auth-proxy-7dc95696d9-wz9vg" Sep 09 02:37:41.530315 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:41.530171 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74f65ba8-8aae-4b5a-9229-0813168e1063-tmp\") pod \"kube-auth-proxy-7dc95696d9-wz9vg\" (UID: \"74f65ba8-8aae-4b5a-9229-0813168e1063\") " pod="openshift-ingress/kube-auth-proxy-7dc95696d9-wz9vg" Sep 09 02:37:41.530315 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:41.530193 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxcws\" (UniqueName: \"kubernetes.io/projected/74f65ba8-8aae-4b5a-9229-0813168e1063-kube-api-access-fxcws\") pod \"kube-auth-proxy-7dc95696d9-wz9vg\" (UID: \"74f65ba8-8aae-4b5a-9229-0813168e1063\") " pod="openshift-ingress/kube-auth-proxy-7dc95696d9-wz9vg" Sep 09 02:37:41.532238 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:41.532209 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:41.630976 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:41.630941 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f65ba8-8aae-4b5a-9229-0813168e1063-tls-certs\") pod \"kube-auth-proxy-7dc95696d9-wz9vg\" (UID: \"74f65ba8-8aae-4b5a-9229-0813168e1063\") " pod="openshift-ingress/kube-auth-proxy-7dc95696d9-wz9vg" Sep 09 02:37:41.630976 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:41.630982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74f65ba8-8aae-4b5a-9229-0813168e1063-tmp\") pod \"kube-auth-proxy-7dc95696d9-wz9vg\" (UID: \"74f65ba8-8aae-4b5a-9229-0813168e1063\") " pod="openshift-ingress/kube-auth-proxy-7dc95696d9-wz9vg" Sep 09 02:37:41.631213 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:41.631012 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxcws\" (UniqueName: \"kubernetes.io/projected/74f65ba8-8aae-4b5a-9229-0813168e1063-kube-api-access-fxcws\") pod \"kube-auth-proxy-7dc95696d9-wz9vg\" (UID: \"74f65ba8-8aae-4b5a-9229-0813168e1063\") " pod="openshift-ingress/kube-auth-proxy-7dc95696d9-wz9vg" Sep 09 02:37:41.633253 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:41.633229 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f65ba8-8aae-4b5a-9229-0813168e1063-tls-certs\") pod \"kube-auth-proxy-7dc95696d9-wz9vg\" (UID: \"74f65ba8-8aae-4b5a-9229-0813168e1063\") " pod="openshift-ingress/kube-auth-proxy-7dc95696d9-wz9vg" Sep 09 02:37:41.633394 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:41.633228 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74f65ba8-8aae-4b5a-9229-0813168e1063-tmp\") pod \"kube-auth-proxy-7dc95696d9-wz9vg\" (UID: \"74f65ba8-8aae-4b5a-9229-0813168e1063\") " pod="openshift-ingress/kube-auth-proxy-7dc95696d9-wz9vg" Sep 09 02:37:41.642420 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:41.642397 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxcws\" (UniqueName: \"kubernetes.io/projected/74f65ba8-8aae-4b5a-9229-0813168e1063-kube-api-access-fxcws\") pod \"kube-auth-proxy-7dc95696d9-wz9vg\" (UID: \"74f65ba8-8aae-4b5a-9229-0813168e1063\") " pod="openshift-ingress/kube-auth-proxy-7dc95696d9-wz9vg" Sep 09 02:37:41.781728 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:41.781643 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7dc95696d9-wz9vg" Sep 09 02:37:42.059498 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:42.059473 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7dc95696d9-wz9vg"] Sep 09 02:37:42.061547 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:37:42.061509 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f65ba8_8aae_4b5a_9229_0813168e1063.slice/crio-1140448c33e49ed4e7475c466496b19c8e48a1a0f3cb37b4bdd9a87fc65d06eb WatchSource:0}: Error finding container 1140448c33e49ed4e7475c466496b19c8e48a1a0f3cb37b4bdd9a87fc65d06eb: Status 404 returned error can't find the container with id 1140448c33e49ed4e7475c466496b19c8e48a1a0f3cb37b4bdd9a87fc65d06eb Sep 09 02:37:42.505151 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:42.505112 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7dc95696d9-wz9vg" event={"ID":"74f65ba8-8aae-4b5a-9229-0813168e1063","Type":"ContainerStarted","Data":"1140448c33e49ed4e7475c466496b19c8e48a1a0f3cb37b4bdd9a87fc65d06eb"} Sep 09 02:37:43.033197 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:43.033158 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:43.127420 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:43.127351 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:44.363707 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:44.363668 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:45.523860 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:45.523807 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7dc95696d9-wz9vg" event={"ID":"74f65ba8-8aae-4b5a-9229-0813168e1063","Type":"ContainerStarted","Data":"2d686dfb67c50e820014f43315c426783d70def7553d80274aef24b4605cfc1e"} Sep 09 02:37:45.545685 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:45.545634 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7dc95696d9-wz9vg" podStartSLOduration=1.75976196 podStartE2EDuration="4.545619311s" totalImagesPullingTime="2.785857351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:37:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:37:45.544302847 +0000 UTC m=+453.674544941" watchObservedRunningTime="2026-09-09 02:37:45.545619311 +0000 UTC m=+453.675861407" Sep 09 02:37:48.160805 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:48.160764 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:48.261176 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:48.261147 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:48.941876 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:48.941843 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:49.171262 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:49.171228 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:49.382751 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:49.382718 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:51.949503 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:51.949466 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:52.049704 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:52.049666 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:52.150406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:52.150371 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:52.570034 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:52.569997 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:37:52.884064 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:37:52.884036 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:38:02.400072 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:38:02.400027 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:38:02.589935 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:38:02.589906 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:38:02.799261 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:38:02.799177 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:38:02.915208 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:38:02.915165 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:38:03.898915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:38:03.898865 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:38:04.092998 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:38:04.092968 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:38:44.381039 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:38:44.381007 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:38:53.103525 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:38:53.103483 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:38:53.200957 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:38:53.200919 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:08.378709 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:08.378626 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:09.174856 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:09.174809 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:12.189057 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:12.189024 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:12.449425 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:12.449331 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:12.542384 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:12.542348 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:12.695491 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:12.695454 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:13.035211 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:13.035172 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:13.267141 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:13.267102 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:14.870428 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:14.870395 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:14.970571 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:14.970529 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:16.609512 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:16.609481 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:16.930387 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:16.930350 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:18.862262 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:18.862230 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:20.727496 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:20.727462 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:21.630252 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:21.630178 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:21.741591 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:21.741564 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:22.040676 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:22.040594 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:22.192662 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:22.192631 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:27.819449 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:27.819416 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:27.966637 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:27.966601 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:28.064396 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:28.064364 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:28.454060 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:28.454017 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:28.534246 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:28.534217 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:28.613879 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:28.613846 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:28.715904 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:28.715792 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:33.356864 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:33.356836 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:33.454976 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:33.454946 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:37.727759 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:37.727725 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-82pv5"] Sep 09 02:39:37.731198 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:37.731178 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-82pv5" Sep 09 02:39:37.734978 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:37.734948 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 09 02:39:37.735084 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:37.735016 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 09 02:39:37.739397 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:37.739380 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bh6dl\"" Sep 09 02:39:37.746005 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:37.745984 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-82pv5"] Sep 09 02:39:37.845972 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:37.845941 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32944e78-c943-48f9-a2c8-75064853253c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-82pv5\" (UID: \"32944e78-c943-48f9-a2c8-75064853253c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-82pv5" Sep 09 02:39:37.846133 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:37.845987 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgwp5\" (UniqueName: \"kubernetes.io/projected/32944e78-c943-48f9-a2c8-75064853253c-kube-api-access-hgwp5\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-82pv5\" (UID: \"32944e78-c943-48f9-a2c8-75064853253c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-82pv5" Sep 09 02:39:37.946737 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:37.946704 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32944e78-c943-48f9-a2c8-75064853253c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-82pv5\" (UID: \"32944e78-c943-48f9-a2c8-75064853253c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-82pv5" Sep 09 02:39:37.946921 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:37.946750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgwp5\" (UniqueName: \"kubernetes.io/projected/32944e78-c943-48f9-a2c8-75064853253c-kube-api-access-hgwp5\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-82pv5\" (UID: \"32944e78-c943-48f9-a2c8-75064853253c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-82pv5" Sep 09 02:39:37.947098 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:37.947078 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32944e78-c943-48f9-a2c8-75064853253c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-82pv5\" (UID: \"32944e78-c943-48f9-a2c8-75064853253c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-82pv5" Sep 09 02:39:37.959105 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:37.959086 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgwp5\" (UniqueName: \"kubernetes.io/projected/32944e78-c943-48f9-a2c8-75064853253c-kube-api-access-hgwp5\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-82pv5\" (UID: \"32944e78-c943-48f9-a2c8-75064853253c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-82pv5" Sep 09 02:39:38.042226 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:38.042157 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-82pv5" Sep 09 02:39:38.235756 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:38.235708 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:38.335969 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:39:38.335939 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32944e78_c943_48f9_a2c8_75064853253c.slice/crio-c8edd69b5cd21df2d273602b94fc41a63008940584e2906709d3a82c07c52d1a WatchSource:0}: Error finding container c8edd69b5cd21df2d273602b94fc41a63008940584e2906709d3a82c07c52d1a: Status 404 returned error can't find the container with id c8edd69b5cd21df2d273602b94fc41a63008940584e2906709d3a82c07c52d1a Sep 09 02:39:38.336239 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:38.336216 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-82pv5"] Sep 09 02:39:38.341503 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:38.341467 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:38.914761 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:38.914721 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-82pv5"] Sep 09 02:39:38.919324 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:38.919297 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-82pv5"] Sep 09 02:39:41.124330 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:41.124295 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:41.232861 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:41.232826 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:42.099881 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:42.099793 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:42.256141 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:42.256107 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:44.087367 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:44.087325 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-82pv5" podUID="32944e78-c943-48f9-a2c8-75064853253c" containerName="manager" containerID="cri-o://0d79c489112bd9a26a5b038d08110b7a89a54d6e5393a5120661325e34477dc1" gracePeriod=2 Sep 09 02:39:44.089559 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:39:44.089533 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-82pv5\" is forbidden: User \"system:node:ip-10-0-137-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-37.ec2.internal' and this object" podUID="32944e78-c943-48f9-a2c8-75064853253c" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-82pv5" Sep 09 02:39:44.294078 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:44.294053 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-82pv5" Sep 09 02:39:44.296447 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:39:44.296426 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-82pv5\" is forbidden: User \"system:node:ip-10-0-137-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-37.ec2.internal' and this object" podUID="32944e78-c943-48f9-a2c8-75064853253c" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-82pv5" Sep 09 02:39:44.409385 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:44.409352 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32944e78-c943-48f9-a2c8-75064853253c-extensions-socket-volume\") pod \"32944e78-c943-48f9-a2c8-75064853253c\" (UID: \"32944e78-c943-48f9-a2c8-75064853253c\") " Sep 09 02:39:44.409550 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:44.409442 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgwp5\" (UniqueName: \"kubernetes.io/projected/32944e78-c943-48f9-a2c8-75064853253c-kube-api-access-hgwp5\") pod \"32944e78-c943-48f9-a2c8-75064853253c\" (UID: \"32944e78-c943-48f9-a2c8-75064853253c\") " Sep 09 02:39:44.409652 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:44.409603 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32944e78-c943-48f9-a2c8-75064853253c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "32944e78-c943-48f9-a2c8-75064853253c" (UID: "32944e78-c943-48f9-a2c8-75064853253c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:39:44.411641 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:44.411617 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32944e78-c943-48f9-a2c8-75064853253c-kube-api-access-hgwp5" (OuterVolumeSpecName: "kube-api-access-hgwp5") pod "32944e78-c943-48f9-a2c8-75064853253c" (UID: "32944e78-c943-48f9-a2c8-75064853253c"). InnerVolumeSpecName "kube-api-access-hgwp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:39:44.510680 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:44.510637 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32944e78-c943-48f9-a2c8-75064853253c-extensions-socket-volume\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:39:44.510680 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:44.510674 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgwp5\" (UniqueName: \"kubernetes.io/projected/32944e78-c943-48f9-a2c8-75064853253c-kube-api-access-hgwp5\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:39:45.099511 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:45.099474 2577 generic.go:356] "Generic (PLEG): container finished" podID="32944e78-c943-48f9-a2c8-75064853253c" containerID="0d79c489112bd9a26a5b038d08110b7a89a54d6e5393a5120661325e34477dc1" exitCode=2 Sep 09 02:39:45.100051 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:45.099518 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-82pv5" Sep 09 02:39:45.100051 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:45.099546 2577 scope.go:117] "RemoveContainer" containerID="0d79c489112bd9a26a5b038d08110b7a89a54d6e5393a5120661325e34477dc1" Sep 09 02:39:45.102034 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:39:45.102007 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-82pv5\" is forbidden: User \"system:node:ip-10-0-137-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-37.ec2.internal' and this object" podUID="32944e78-c943-48f9-a2c8-75064853253c" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-82pv5" Sep 09 02:39:45.108719 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:45.108702 2577 scope.go:117] "RemoveContainer" containerID="0d79c489112bd9a26a5b038d08110b7a89a54d6e5393a5120661325e34477dc1" Sep 09 02:39:45.108983 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:39:45.108965 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d79c489112bd9a26a5b038d08110b7a89a54d6e5393a5120661325e34477dc1\": container with ID starting with 0d79c489112bd9a26a5b038d08110b7a89a54d6e5393a5120661325e34477dc1 not found: ID does not exist" containerID="0d79c489112bd9a26a5b038d08110b7a89a54d6e5393a5120661325e34477dc1" Sep 09 02:39:45.109033 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:45.108991 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d79c489112bd9a26a5b038d08110b7a89a54d6e5393a5120661325e34477dc1"} err="failed to get container status \"0d79c489112bd9a26a5b038d08110b7a89a54d6e5393a5120661325e34477dc1\": rpc error: code = NotFound desc = could not find container \"0d79c489112bd9a26a5b038d08110b7a89a54d6e5393a5120661325e34477dc1\": container with ID starting with 0d79c489112bd9a26a5b038d08110b7a89a54d6e5393a5120661325e34477dc1 not found: ID does not exist" Sep 09 02:39:45.110416 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:39:45.110398 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-82pv5\" is forbidden: User \"system:node:ip-10-0-137-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-37.ec2.internal' and this object" podUID="32944e78-c943-48f9-a2c8-75064853253c" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-82pv5" Sep 09 02:39:46.309493 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:46.309462 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32944e78-c943-48f9-a2c8-75064853253c" path="/var/lib/kubelet/pods/32944e78-c943-48f9-a2c8-75064853253c/volumes" Sep 09 02:39:46.319529 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:46.319501 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:46.419342 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:46.419310 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:46.899674 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:46.899640 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:46.997932 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:46.997900 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:47.512505 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:47.512476 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:47.611094 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:47.611059 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:55.708300 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:55.708265 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:55.808899 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:55.808866 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:39:59.905245 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:39:59.905209 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:00.007525 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:00.007494 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:03.084692 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:03.084659 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:03.164861 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:03.164810 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:04.101340 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:04.101284 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:04.160436 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:04.160401 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:04.704706 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:04.704673 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:04.804443 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:04.804411 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:08.372316 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:08.372278 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:08.472615 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:08.472580 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:09.701745 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:09.701714 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:09.827271 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:09.827241 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:09.927741 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:09.927714 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:09.998582 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:09.998489 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:10.103770 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:10.103745 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:10.204008 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:10.203976 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:12.284596 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:12.284563 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/2.log" Sep 09 02:40:12.285062 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:12.284801 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/2.log" Sep 09 02:40:16.028559 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:16.028519 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:16.090490 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:16.090454 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:20.777200 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:20.777165 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:20.882991 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:20.882952 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:23.102341 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.102304 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:23.103043 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.103023 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-84l6h"] Sep 09 02:40:23.103388 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.103376 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32944e78-c943-48f9-a2c8-75064853253c" containerName="manager" Sep 09 02:40:23.103426 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.103390 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="32944e78-c943-48f9-a2c8-75064853253c" containerName="manager" Sep 09 02:40:23.103460 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.103454 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="32944e78-c943-48f9-a2c8-75064853253c" containerName="manager" Sep 09 02:40:23.106548 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.106527 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-84l6h" Sep 09 02:40:23.109396 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.109369 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 09 02:40:23.109513 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.109410 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xm4tk\"" Sep 09 02:40:23.110515 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.110497 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 09 02:40:23.110632 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.110588 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 09 02:40:23.119749 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.119721 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-84l6h"] Sep 09 02:40:23.197096 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.197064 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-84l6h"] Sep 09 02:40:23.197941 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.197910 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:23.240950 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.240915 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f236d8c2-fcc9-4487-87dd-a5cb37c533bb-config-file\") pod \"limitador-limitador-8588749977-84l6h\" (UID: \"f236d8c2-fcc9-4487-87dd-a5cb37c533bb\") " pod="kuadrant-system/limitador-limitador-8588749977-84l6h" Sep 09 02:40:23.241100 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.240968 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2z4k\" (UniqueName: \"kubernetes.io/projected/f236d8c2-fcc9-4487-87dd-a5cb37c533bb-kube-api-access-h2z4k\") pod \"limitador-limitador-8588749977-84l6h\" (UID: \"f236d8c2-fcc9-4487-87dd-a5cb37c533bb\") " pod="kuadrant-system/limitador-limitador-8588749977-84l6h" Sep 09 02:40:23.296792 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.296761 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:23.341987 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.341951 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f236d8c2-fcc9-4487-87dd-a5cb37c533bb-config-file\") pod \"limitador-limitador-8588749977-84l6h\" (UID: \"f236d8c2-fcc9-4487-87dd-a5cb37c533bb\") " pod="kuadrant-system/limitador-limitador-8588749977-84l6h" Sep 09 02:40:23.342131 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.342000 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2z4k\" (UniqueName: \"kubernetes.io/projected/f236d8c2-fcc9-4487-87dd-a5cb37c533bb-kube-api-access-h2z4k\") pod \"limitador-limitador-8588749977-84l6h\" (UID: \"f236d8c2-fcc9-4487-87dd-a5cb37c533bb\") " pod="kuadrant-system/limitador-limitador-8588749977-84l6h" Sep 09 02:40:23.342574 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.342553 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f236d8c2-fcc9-4487-87dd-a5cb37c533bb-config-file\") pod \"limitador-limitador-8588749977-84l6h\" (UID: \"f236d8c2-fcc9-4487-87dd-a5cb37c533bb\") " pod="kuadrant-system/limitador-limitador-8588749977-84l6h" Sep 09 02:40:23.350622 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.350599 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2z4k\" (UniqueName: \"kubernetes.io/projected/f236d8c2-fcc9-4487-87dd-a5cb37c533bb-kube-api-access-h2z4k\") pod \"limitador-limitador-8588749977-84l6h\" (UID: \"f236d8c2-fcc9-4487-87dd-a5cb37c533bb\") " pod="kuadrant-system/limitador-limitador-8588749977-84l6h" Sep 09 02:40:23.420553 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.420525 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-84l6h" Sep 09 02:40:23.692222 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.692200 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-84l6h"] Sep 09 02:40:23.694244 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:40:23.694212 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf236d8c2_fcc9_4487_87dd_a5cb37c533bb.slice/crio-123156a51a6fc4118874687f342a20573aabd9f04b4356149452e181b3d9892f WatchSource:0}: Error finding container 123156a51a6fc4118874687f342a20573aabd9f04b4356149452e181b3d9892f: Status 404 returned error can't find the container with id 123156a51a6fc4118874687f342a20573aabd9f04b4356149452e181b3d9892f Sep 09 02:40:23.888827 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.888776 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-j2grk"] Sep 09 02:40:23.893305 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.893280 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-j2grk" Sep 09 02:40:23.895982 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.895958 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nlct4\"" Sep 09 02:40:23.899910 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.899887 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-j2grk"] Sep 09 02:40:23.946497 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:23.946388 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:24.040503 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:24.040463 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:24.048115 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:24.048088 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzfc2\" (UniqueName: \"kubernetes.io/projected/cfad3735-4b2b-49bf-859d-232a11b69e15-kube-api-access-zzfc2\") pod \"authorino-7bcfd8f9cd-j2grk\" (UID: \"cfad3735-4b2b-49bf-859d-232a11b69e15\") " pod="kuadrant-system/authorino-7bcfd8f9cd-j2grk" Sep 09 02:40:24.148786 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:24.148748 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzfc2\" (UniqueName: \"kubernetes.io/projected/cfad3735-4b2b-49bf-859d-232a11b69e15-kube-api-access-zzfc2\") pod \"authorino-7bcfd8f9cd-j2grk\" (UID: \"cfad3735-4b2b-49bf-859d-232a11b69e15\") " pod="kuadrant-system/authorino-7bcfd8f9cd-j2grk" Sep 09 02:40:24.158542 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:24.158517 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzfc2\" (UniqueName: \"kubernetes.io/projected/cfad3735-4b2b-49bf-859d-232a11b69e15-kube-api-access-zzfc2\") pod \"authorino-7bcfd8f9cd-j2grk\" (UID: \"cfad3735-4b2b-49bf-859d-232a11b69e15\") " pod="kuadrant-system/authorino-7bcfd8f9cd-j2grk" Sep 09 02:40:24.204746 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:24.204670 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-j2grk" Sep 09 02:40:24.283277 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:24.283247 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-84l6h" event={"ID":"f236d8c2-fcc9-4487-87dd-a5cb37c533bb","Type":"ContainerStarted","Data":"123156a51a6fc4118874687f342a20573aabd9f04b4356149452e181b3d9892f"} Sep 09 02:40:24.473016 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:24.472944 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-j2grk"] Sep 09 02:40:24.476119 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:40:24.476090 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfad3735_4b2b_49bf_859d_232a11b69e15.slice/crio-e138292618590f57cbd22048ffe43bae8f88d0401caccf34cc61766113493b75 WatchSource:0}: Error finding container e138292618590f57cbd22048ffe43bae8f88d0401caccf34cc61766113493b75: Status 404 returned error can't find the container with id e138292618590f57cbd22048ffe43bae8f88d0401caccf34cc61766113493b75 Sep 09 02:40:25.290266 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:25.290234 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-j2grk" event={"ID":"cfad3735-4b2b-49bf-859d-232a11b69e15","Type":"ContainerStarted","Data":"e138292618590f57cbd22048ffe43bae8f88d0401caccf34cc61766113493b75"} Sep 09 02:40:26.297843 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:26.297797 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-84l6h" event={"ID":"f236d8c2-fcc9-4487-87dd-a5cb37c533bb","Type":"ContainerStarted","Data":"5116c6e55c4dce180bcae4ed5f6e4e9a3e45809270bec4d4623968b1c28faeeb"} Sep 09 02:40:26.298296 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:26.297960 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-84l6h" Sep 09 02:40:26.318347 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:26.318303 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-84l6h" podStartSLOduration=1.719847806 podStartE2EDuration="3.318285058s" totalImagesPullingTime="1.598437252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:40:26.315490619 +0000 UTC m=+614.445732739" watchObservedRunningTime="2026-09-09 02:40:26.318285058 +0000 UTC m=+614.448527153" Sep 09 02:40:26.747438 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:26.747404 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-j2grk"] Sep 09 02:40:27.375467 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:27.375440 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:28.309991 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:28.309954 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-j2grk" event={"ID":"cfad3735-4b2b-49bf-859d-232a11b69e15","Type":"ContainerStarted","Data":"95913c2426ff44f9be0b741bd0042ce981b9d2656d297c214e6344833ca381f9"} Sep 09 02:40:28.310168 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:28.310001 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-j2grk" podUID="cfad3735-4b2b-49bf-859d-232a11b69e15" containerName="authorino" containerID="cri-o://95913c2426ff44f9be0b741bd0042ce981b9d2656d297c214e6344833ca381f9" gracePeriod=30 Sep 09 02:40:28.326248 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:28.326204 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-j2grk" podStartSLOduration=2.44847444 podStartE2EDuration="5.326188967s" totalImagesPullingTime="2.877714527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:40:28.325451356 +0000 UTC m=+616.455693453" watchObservedRunningTime="2026-09-09 02:40:28.326188967 +0000 UTC m=+616.456431063" Sep 09 02:40:28.553188 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:28.553166 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-j2grk" Sep 09 02:40:28.688991 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:28.688959 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzfc2\" (UniqueName: \"kubernetes.io/projected/cfad3735-4b2b-49bf-859d-232a11b69e15-kube-api-access-zzfc2\") pod \"cfad3735-4b2b-49bf-859d-232a11b69e15\" (UID: \"cfad3735-4b2b-49bf-859d-232a11b69e15\") " Sep 09 02:40:28.691026 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:28.690999 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfad3735-4b2b-49bf-859d-232a11b69e15-kube-api-access-zzfc2" (OuterVolumeSpecName: "kube-api-access-zzfc2") pod "cfad3735-4b2b-49bf-859d-232a11b69e15" (UID: "cfad3735-4b2b-49bf-859d-232a11b69e15"). InnerVolumeSpecName "kube-api-access-zzfc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:40:28.790207 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:28.790171 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzfc2\" (UniqueName: \"kubernetes.io/projected/cfad3735-4b2b-49bf-859d-232a11b69e15-kube-api-access-zzfc2\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:40:29.316897 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:29.316858 2577 generic.go:356] "Generic (PLEG): container finished" podID="cfad3735-4b2b-49bf-859d-232a11b69e15" containerID="95913c2426ff44f9be0b741bd0042ce981b9d2656d297c214e6344833ca381f9" exitCode=0 Sep 09 02:40:29.317095 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:29.316918 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-j2grk" Sep 09 02:40:29.317095 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:29.316947 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-j2grk" event={"ID":"cfad3735-4b2b-49bf-859d-232a11b69e15","Type":"ContainerDied","Data":"95913c2426ff44f9be0b741bd0042ce981b9d2656d297c214e6344833ca381f9"} Sep 09 02:40:29.317095 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:29.316991 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-j2grk" event={"ID":"cfad3735-4b2b-49bf-859d-232a11b69e15","Type":"ContainerDied","Data":"e138292618590f57cbd22048ffe43bae8f88d0401caccf34cc61766113493b75"} Sep 09 02:40:29.317095 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:29.317009 2577 scope.go:117] "RemoveContainer" containerID="95913c2426ff44f9be0b741bd0042ce981b9d2656d297c214e6344833ca381f9" Sep 09 02:40:29.325993 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:29.325804 2577 scope.go:117] "RemoveContainer" containerID="95913c2426ff44f9be0b741bd0042ce981b9d2656d297c214e6344833ca381f9" Sep 09 02:40:29.326127 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:40:29.326109 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95913c2426ff44f9be0b741bd0042ce981b9d2656d297c214e6344833ca381f9\": container with ID starting with 95913c2426ff44f9be0b741bd0042ce981b9d2656d297c214e6344833ca381f9 not found: ID does not exist" containerID="95913c2426ff44f9be0b741bd0042ce981b9d2656d297c214e6344833ca381f9" Sep 09 02:40:29.326170 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:29.326136 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95913c2426ff44f9be0b741bd0042ce981b9d2656d297c214e6344833ca381f9"} err="failed to get container status \"95913c2426ff44f9be0b741bd0042ce981b9d2656d297c214e6344833ca381f9\": rpc error: code = NotFound desc = could not find container \"95913c2426ff44f9be0b741bd0042ce981b9d2656d297c214e6344833ca381f9\": container with ID starting with 95913c2426ff44f9be0b741bd0042ce981b9d2656d297c214e6344833ca381f9 not found: ID does not exist" Sep 09 02:40:29.340066 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:29.340043 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-j2grk"] Sep 09 02:40:29.345327 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:29.345305 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-j2grk"] Sep 09 02:40:30.310050 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:30.310021 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfad3735-4b2b-49bf-859d-232a11b69e15" path="/var/lib/kubelet/pods/cfad3735-4b2b-49bf-859d-232a11b69e15/volumes" Sep 09 02:40:36.759890 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:36.759791 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-84l6h"] Sep 09 02:40:36.760367 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:36.760042 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-84l6h" podUID="f236d8c2-fcc9-4487-87dd-a5cb37c533bb" containerName="limitador" containerID="cri-o://5116c6e55c4dce180bcae4ed5f6e4e9a3e45809270bec4d4623968b1c28faeeb" gracePeriod=30 Sep 09 02:40:36.760758 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:36.760740 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-84l6h" Sep 09 02:40:37.302555 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:37.302527 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-84l6h" Sep 09 02:40:37.354372 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:37.354283 2577 generic.go:356] "Generic (PLEG): container finished" podID="f236d8c2-fcc9-4487-87dd-a5cb37c533bb" containerID="5116c6e55c4dce180bcae4ed5f6e4e9a3e45809270bec4d4623968b1c28faeeb" exitCode=0 Sep 09 02:40:37.354372 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:37.354328 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-84l6h" Sep 09 02:40:37.354575 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:37.354331 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-84l6h" event={"ID":"f236d8c2-fcc9-4487-87dd-a5cb37c533bb","Type":"ContainerDied","Data":"5116c6e55c4dce180bcae4ed5f6e4e9a3e45809270bec4d4623968b1c28faeeb"} Sep 09 02:40:37.354575 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:37.354461 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-84l6h" event={"ID":"f236d8c2-fcc9-4487-87dd-a5cb37c533bb","Type":"ContainerDied","Data":"123156a51a6fc4118874687f342a20573aabd9f04b4356149452e181b3d9892f"} Sep 09 02:40:37.354575 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:37.354494 2577 scope.go:117] "RemoveContainer" containerID="5116c6e55c4dce180bcae4ed5f6e4e9a3e45809270bec4d4623968b1c28faeeb" Sep 09 02:40:37.362579 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:37.362561 2577 scope.go:117] "RemoveContainer" containerID="5116c6e55c4dce180bcae4ed5f6e4e9a3e45809270bec4d4623968b1c28faeeb" Sep 09 02:40:37.362925 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:40:37.362903 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5116c6e55c4dce180bcae4ed5f6e4e9a3e45809270bec4d4623968b1c28faeeb\": container with ID starting with 5116c6e55c4dce180bcae4ed5f6e4e9a3e45809270bec4d4623968b1c28faeeb not found: ID does not exist" containerID="5116c6e55c4dce180bcae4ed5f6e4e9a3e45809270bec4d4623968b1c28faeeb" Sep 09 02:40:37.362980 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:37.362935 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5116c6e55c4dce180bcae4ed5f6e4e9a3e45809270bec4d4623968b1c28faeeb"} err="failed to get container status \"5116c6e55c4dce180bcae4ed5f6e4e9a3e45809270bec4d4623968b1c28faeeb\": rpc error: code = NotFound desc = could not find container \"5116c6e55c4dce180bcae4ed5f6e4e9a3e45809270bec4d4623968b1c28faeeb\": container with ID starting with 5116c6e55c4dce180bcae4ed5f6e4e9a3e45809270bec4d4623968b1c28faeeb not found: ID does not exist" Sep 09 02:40:37.466954 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:37.466919 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2z4k\" (UniqueName: \"kubernetes.io/projected/f236d8c2-fcc9-4487-87dd-a5cb37c533bb-kube-api-access-h2z4k\") pod \"f236d8c2-fcc9-4487-87dd-a5cb37c533bb\" (UID: \"f236d8c2-fcc9-4487-87dd-a5cb37c533bb\") " Sep 09 02:40:37.466954 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:37.466956 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f236d8c2-fcc9-4487-87dd-a5cb37c533bb-config-file\") pod \"f236d8c2-fcc9-4487-87dd-a5cb37c533bb\" (UID: \"f236d8c2-fcc9-4487-87dd-a5cb37c533bb\") " Sep 09 02:40:37.467371 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:37.467347 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f236d8c2-fcc9-4487-87dd-a5cb37c533bb-config-file" (OuterVolumeSpecName: "config-file") pod "f236d8c2-fcc9-4487-87dd-a5cb37c533bb" (UID: "f236d8c2-fcc9-4487-87dd-a5cb37c533bb"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:40:37.469019 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:37.468998 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f236d8c2-fcc9-4487-87dd-a5cb37c533bb-kube-api-access-h2z4k" (OuterVolumeSpecName: "kube-api-access-h2z4k") pod "f236d8c2-fcc9-4487-87dd-a5cb37c533bb" (UID: "f236d8c2-fcc9-4487-87dd-a5cb37c533bb"). InnerVolumeSpecName "kube-api-access-h2z4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:40:37.567592 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:37.567563 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2z4k\" (UniqueName: \"kubernetes.io/projected/f236d8c2-fcc9-4487-87dd-a5cb37c533bb-kube-api-access-h2z4k\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:40:37.567592 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:37.567588 2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f236d8c2-fcc9-4487-87dd-a5cb37c533bb-config-file\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:40:37.680827 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:37.680794 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-84l6h"] Sep 09 02:40:37.684945 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:37.684922 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-84l6h"] Sep 09 02:40:38.309846 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:38.309798 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f236d8c2-fcc9-4487-87dd-a5cb37c533bb" path="/var/lib/kubelet/pods/f236d8c2-fcc9-4487-87dd-a5cb37c533bb/volumes" Sep 09 02:40:42.316518 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:42.316468 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:42.754744 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:42.754704 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-78hwb"] Sep 09 02:40:42.755184 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:42.755167 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f236d8c2-fcc9-4487-87dd-a5cb37c533bb" containerName="limitador" Sep 09 02:40:42.755241 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:42.755186 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f236d8c2-fcc9-4487-87dd-a5cb37c533bb" containerName="limitador" Sep 09 02:40:42.755241 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:42.755209 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfad3735-4b2b-49bf-859d-232a11b69e15" containerName="authorino" Sep 09 02:40:42.755241 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:42.755219 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfad3735-4b2b-49bf-859d-232a11b69e15" containerName="authorino" Sep 09 02:40:42.755393 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:42.755313 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f236d8c2-fcc9-4487-87dd-a5cb37c533bb" containerName="limitador" Sep 09 02:40:42.755393 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:42.755331 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfad3735-4b2b-49bf-859d-232a11b69e15" containerName="authorino" Sep 09 02:40:42.760264 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:42.760242 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-78hwb" Sep 09 02:40:42.763233 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:42.763210 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 09 02:40:42.763366 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:42.763248 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 09 02:40:42.764261 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:42.764238 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-qkd7j\"" Sep 09 02:40:42.764394 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:42.764275 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 02:40:42.771412 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:42.770613 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-78hwb"] Sep 09 02:40:42.817330 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:42.817292 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4ed93ff6-dddd-4e46-9eee-850b42de02f3-data\") pod \"postgres-69fb4594cf-78hwb\" (UID: \"4ed93ff6-dddd-4e46-9eee-850b42de02f3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-78hwb" Sep 09 02:40:42.817330 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:42.817334 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz8d6\" (UniqueName: \"kubernetes.io/projected/4ed93ff6-dddd-4e46-9eee-850b42de02f3-kube-api-access-sz8d6\") pod \"postgres-69fb4594cf-78hwb\" (UID: \"4ed93ff6-dddd-4e46-9eee-850b42de02f3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-78hwb" Sep 09 02:40:42.918590 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:42.918553 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4ed93ff6-dddd-4e46-9eee-850b42de02f3-data\") pod \"postgres-69fb4594cf-78hwb\" (UID: \"4ed93ff6-dddd-4e46-9eee-850b42de02f3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-78hwb" Sep 09 02:40:42.918759 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:42.918598 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz8d6\" (UniqueName: \"kubernetes.io/projected/4ed93ff6-dddd-4e46-9eee-850b42de02f3-kube-api-access-sz8d6\") pod \"postgres-69fb4594cf-78hwb\" (UID: \"4ed93ff6-dddd-4e46-9eee-850b42de02f3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-78hwb" Sep 09 02:40:42.918954 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:42.918934 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4ed93ff6-dddd-4e46-9eee-850b42de02f3-data\") pod \"postgres-69fb4594cf-78hwb\" (UID: \"4ed93ff6-dddd-4e46-9eee-850b42de02f3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-78hwb" Sep 09 02:40:42.927768 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:42.927743 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz8d6\" (UniqueName: \"kubernetes.io/projected/4ed93ff6-dddd-4e46-9eee-850b42de02f3-kube-api-access-sz8d6\") pod \"postgres-69fb4594cf-78hwb\" (UID: \"4ed93ff6-dddd-4e46-9eee-850b42de02f3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-78hwb" Sep 09 02:40:43.075669 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:43.075582 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-78hwb" Sep 09 02:40:43.346913 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:43.346812 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-78hwb"] Sep 09 02:40:43.350679 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:40:43.350649 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed93ff6_dddd_4e46_9eee_850b42de02f3.slice/crio-dbd07f9a52084c9b68f45a0e030ad4ef85d3e420176523f438425ea914af41b6 WatchSource:0}: Error finding container dbd07f9a52084c9b68f45a0e030ad4ef85d3e420176523f438425ea914af41b6: Status 404 returned error can't find the container with id dbd07f9a52084c9b68f45a0e030ad4ef85d3e420176523f438425ea914af41b6 Sep 09 02:40:43.383134 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:43.383100 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-78hwb" event={"ID":"4ed93ff6-dddd-4e46-9eee-850b42de02f3","Type":"ContainerStarted","Data":"dbd07f9a52084c9b68f45a0e030ad4ef85d3e420176523f438425ea914af41b6"} Sep 09 02:40:43.681477 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:43.681440 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:49.415477 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:49.415436 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-78hwb" event={"ID":"4ed93ff6-dddd-4e46-9eee-850b42de02f3","Type":"ContainerStarted","Data":"2ad6185666a558a879fb988ee3f37a8602ef302f57574907cafed0888eccf0a9"} Sep 09 02:40:49.415883 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:49.415507 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-78hwb" Sep 09 02:40:49.433742 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:49.433692 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-78hwb" podStartSLOduration=1.587190841 podStartE2EDuration="7.433675412s" totalImagesPullingTime="5.846484571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:40:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:40:49.433253476 +0000 UTC m=+637.563495573" watchObservedRunningTime="2026-09-09 02:40:49.433675412 +0000 UTC m=+637.563917507" Sep 09 02:40:55.450463 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:55.450431 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-78hwb" Sep 09 02:40:55.775955 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:55.775871 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:56.004269 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.004237 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-7476q"] Sep 09 02:40:56.010378 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.010357 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-7476q" Sep 09 02:40:56.013165 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.013145 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 09 02:40:56.014321 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.014296 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nlct4\"" Sep 09 02:40:56.014452 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.014346 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 09 02:40:56.017105 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.017082 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-7476q"] Sep 09 02:40:56.129868 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.129836 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:40:56.142967 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.142940 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgsn5\" (UniqueName: \"kubernetes.io/projected/472606ea-ddc5-42c0-a084-aa2ac7e432cf-kube-api-access-xgsn5\") pod \"authorino-557f876f69-7476q\" (UID: \"472606ea-ddc5-42c0-a084-aa2ac7e432cf\") " pod="kuadrant-system/authorino-557f876f69-7476q" Sep 09 02:40:56.210636 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.210604 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-7476q"] Sep 09 02:40:56.210890 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:40:56.210869 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xgsn5], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-7476q" podUID="472606ea-ddc5-42c0-a084-aa2ac7e432cf" Sep 09 02:40:56.239175 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.239143 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9kmgj"] Sep 09 02:40:56.243503 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.243485 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-9kmgj" Sep 09 02:40:56.243630 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.243609 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgsn5\" (UniqueName: \"kubernetes.io/projected/472606ea-ddc5-42c0-a084-aa2ac7e432cf-kube-api-access-xgsn5\") pod \"authorino-557f876f69-7476q\" (UID: \"472606ea-ddc5-42c0-a084-aa2ac7e432cf\") " pod="kuadrant-system/authorino-557f876f69-7476q" Sep 09 02:40:56.246474 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.246457 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 09 02:40:56.252919 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.252898 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9kmgj"] Sep 09 02:40:56.254955 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.254937 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgsn5\" (UniqueName: \"kubernetes.io/projected/472606ea-ddc5-42c0-a084-aa2ac7e432cf-kube-api-access-xgsn5\") pod \"authorino-557f876f69-7476q\" (UID: \"472606ea-ddc5-42c0-a084-aa2ac7e432cf\") " pod="kuadrant-system/authorino-557f876f69-7476q" Sep 09 02:40:56.274416 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.274388 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9kmgj"] Sep 09 02:40:56.274668 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:40:56.274649 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-jsrqk tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-jsrqk tls-cert]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-9kmgj" podUID="4fc43f4a-784a-4f94-9744-bd025c96edee" Sep 09 02:40:56.311342 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.311314 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-758d56c575-xr2ml"] Sep 09 02:40:56.315538 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.315523 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-758d56c575-xr2ml" Sep 09 02:40:56.326227 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.326203 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-758d56c575-xr2ml"] Sep 09 02:40:56.344395 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.344374 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4fc43f4a-784a-4f94-9744-bd025c96edee-tls-cert\") pod \"authorino-6d4dfc649f-9kmgj\" (UID: \"4fc43f4a-784a-4f94-9744-bd025c96edee\") " pod="kuadrant-system/authorino-6d4dfc649f-9kmgj" Sep 09 02:40:56.344524 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.344406 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsrqk\" (UniqueName: \"kubernetes.io/projected/4fc43f4a-784a-4f94-9744-bd025c96edee-kube-api-access-jsrqk\") pod \"authorino-6d4dfc649f-9kmgj\" (UID: \"4fc43f4a-784a-4f94-9744-bd025c96edee\") " pod="kuadrant-system/authorino-6d4dfc649f-9kmgj" Sep 09 02:40:56.445762 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.445674 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bc5n\" (UniqueName: \"kubernetes.io/projected/a58cef93-7d4f-4149-aa55-66d78624cf75-kube-api-access-8bc5n\") pod \"authorino-758d56c575-xr2ml\" (UID: \"a58cef93-7d4f-4149-aa55-66d78624cf75\") " pod="kuadrant-system/authorino-758d56c575-xr2ml" Sep 09 02:40:56.445762 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.445758 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4fc43f4a-784a-4f94-9744-bd025c96edee-tls-cert\") pod \"authorino-6d4dfc649f-9kmgj\" (UID: \"4fc43f4a-784a-4f94-9744-bd025c96edee\") " pod="kuadrant-system/authorino-6d4dfc649f-9kmgj" Sep 09 02:40:56.445986 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.445779 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsrqk\" (UniqueName: \"kubernetes.io/projected/4fc43f4a-784a-4f94-9744-bd025c96edee-kube-api-access-jsrqk\") pod \"authorino-6d4dfc649f-9kmgj\" (UID: \"4fc43f4a-784a-4f94-9744-bd025c96edee\") " pod="kuadrant-system/authorino-6d4dfc649f-9kmgj" Sep 09 02:40:56.445986 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.445794 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a58cef93-7d4f-4149-aa55-66d78624cf75-tls-cert\") pod \"authorino-758d56c575-xr2ml\" (UID: \"a58cef93-7d4f-4149-aa55-66d78624cf75\") " pod="kuadrant-system/authorino-758d56c575-xr2ml" Sep 09 02:40:56.449315 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.449285 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4fc43f4a-784a-4f94-9744-bd025c96edee-tls-cert\") pod \"authorino-6d4dfc649f-9kmgj\" (UID: \"4fc43f4a-784a-4f94-9744-bd025c96edee\") " pod="kuadrant-system/authorino-6d4dfc649f-9kmgj" Sep 09 02:40:56.455537 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.455508 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsrqk\" (UniqueName: \"kubernetes.io/projected/4fc43f4a-784a-4f94-9744-bd025c96edee-kube-api-access-jsrqk\") pod \"authorino-6d4dfc649f-9kmgj\" (UID: \"4fc43f4a-784a-4f94-9744-bd025c96edee\") " pod="kuadrant-system/authorino-6d4dfc649f-9kmgj" Sep 09 02:40:56.456426 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.456410 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-9kmgj" Sep 09 02:40:56.456504 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.456413 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-7476q" Sep 09 02:40:56.474517 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.474494 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-9kmgj" Sep 09 02:40:56.477880 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.477861 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-7476q" Sep 09 02:40:56.547360 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.547331 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4fc43f4a-784a-4f94-9744-bd025c96edee-tls-cert\") pod \"4fc43f4a-784a-4f94-9744-bd025c96edee\" (UID: \"4fc43f4a-784a-4f94-9744-bd025c96edee\") " Sep 09 02:40:56.547547 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.547392 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgsn5\" (UniqueName: \"kubernetes.io/projected/472606ea-ddc5-42c0-a084-aa2ac7e432cf-kube-api-access-xgsn5\") pod \"472606ea-ddc5-42c0-a084-aa2ac7e432cf\" (UID: \"472606ea-ddc5-42c0-a084-aa2ac7e432cf\") " Sep 09 02:40:56.547547 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.547414 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsrqk\" (UniqueName: \"kubernetes.io/projected/4fc43f4a-784a-4f94-9744-bd025c96edee-kube-api-access-jsrqk\") pod \"4fc43f4a-784a-4f94-9744-bd025c96edee\" (UID: \"4fc43f4a-784a-4f94-9744-bd025c96edee\") " Sep 09 02:40:56.547664 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.547595 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bc5n\" (UniqueName: \"kubernetes.io/projected/a58cef93-7d4f-4149-aa55-66d78624cf75-kube-api-access-8bc5n\") pod \"authorino-758d56c575-xr2ml\" (UID: \"a58cef93-7d4f-4149-aa55-66d78624cf75\") " pod="kuadrant-system/authorino-758d56c575-xr2ml" Sep 09 02:40:56.547719 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.547683 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a58cef93-7d4f-4149-aa55-66d78624cf75-tls-cert\") pod \"authorino-758d56c575-xr2ml\" (UID: \"a58cef93-7d4f-4149-aa55-66d78624cf75\") " pod="kuadrant-system/authorino-758d56c575-xr2ml" Sep 09 02:40:56.549515 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.549487 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fc43f4a-784a-4f94-9744-bd025c96edee-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "4fc43f4a-784a-4f94-9744-bd025c96edee" (UID: "4fc43f4a-784a-4f94-9744-bd025c96edee"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:40:56.549647 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.549524 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472606ea-ddc5-42c0-a084-aa2ac7e432cf-kube-api-access-xgsn5" (OuterVolumeSpecName: "kube-api-access-xgsn5") pod "472606ea-ddc5-42c0-a084-aa2ac7e432cf" (UID: "472606ea-ddc5-42c0-a084-aa2ac7e432cf"). InnerVolumeSpecName "kube-api-access-xgsn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:40:56.549647 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.549604 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc43f4a-784a-4f94-9744-bd025c96edee-kube-api-access-jsrqk" (OuterVolumeSpecName: "kube-api-access-jsrqk") pod "4fc43f4a-784a-4f94-9744-bd025c96edee" (UID: "4fc43f4a-784a-4f94-9744-bd025c96edee"). InnerVolumeSpecName "kube-api-access-jsrqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:40:56.550023 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.550006 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a58cef93-7d4f-4149-aa55-66d78624cf75-tls-cert\") pod \"authorino-758d56c575-xr2ml\" (UID: \"a58cef93-7d4f-4149-aa55-66d78624cf75\") " pod="kuadrant-system/authorino-758d56c575-xr2ml" Sep 09 02:40:56.556389 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.556369 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bc5n\" (UniqueName: \"kubernetes.io/projected/a58cef93-7d4f-4149-aa55-66d78624cf75-kube-api-access-8bc5n\") pod \"authorino-758d56c575-xr2ml\" (UID: \"a58cef93-7d4f-4149-aa55-66d78624cf75\") " pod="kuadrant-system/authorino-758d56c575-xr2ml" Sep 09 02:40:56.624597 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.624559 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-758d56c575-xr2ml" Sep 09 02:40:56.648453 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.648427 2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4fc43f4a-784a-4f94-9744-bd025c96edee-tls-cert\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:40:56.648453 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.648455 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgsn5\" (UniqueName: \"kubernetes.io/projected/472606ea-ddc5-42c0-a084-aa2ac7e432cf-kube-api-access-xgsn5\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:40:56.648620 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.648470 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsrqk\" (UniqueName: \"kubernetes.io/projected/4fc43f4a-784a-4f94-9744-bd025c96edee-kube-api-access-jsrqk\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:40:56.894609 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:56.894411 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-758d56c575-xr2ml"] Sep 09 02:40:56.897387 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:40:56.897358 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda58cef93_7d4f_4149_aa55_66d78624cf75.slice/crio-a2c9ae7db540b80ad846084708b1742f4284f4ef9e7c4be59bb8f797f373a2cc WatchSource:0}: Error finding container a2c9ae7db540b80ad846084708b1742f4284f4ef9e7c4be59bb8f797f373a2cc: Status 404 returned error can't find the container with id a2c9ae7db540b80ad846084708b1742f4284f4ef9e7c4be59bb8f797f373a2cc Sep 09 02:40:57.462908 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:57.462862 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-758d56c575-xr2ml" event={"ID":"a58cef93-7d4f-4149-aa55-66d78624cf75","Type":"ContainerStarted","Data":"a2c9ae7db540b80ad846084708b1742f4284f4ef9e7c4be59bb8f797f373a2cc"} Sep 09 02:40:57.462908 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:57.462905 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-7476q" Sep 09 02:40:57.463459 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:57.462909 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-9kmgj" Sep 09 02:40:57.499381 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:57.499346 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9kmgj"] Sep 09 02:40:57.502766 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:57.502739 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9kmgj"] Sep 09 02:40:57.525941 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:57.525907 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-7476q"] Sep 09 02:40:57.530446 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:57.530421 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-7476q"] Sep 09 02:40:58.310742 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:58.310709 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="472606ea-ddc5-42c0-a084-aa2ac7e432cf" path="/var/lib/kubelet/pods/472606ea-ddc5-42c0-a084-aa2ac7e432cf/volumes" Sep 09 02:40:58.311009 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:58.310995 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fc43f4a-784a-4f94-9744-bd025c96edee" path="/var/lib/kubelet/pods/4fc43f4a-784a-4f94-9744-bd025c96edee/volumes" Sep 09 02:40:58.469176 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:58.469137 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-758d56c575-xr2ml" event={"ID":"a58cef93-7d4f-4149-aa55-66d78624cf75","Type":"ContainerStarted","Data":"ad99b666e1c52c380d04294ae8f615b6ba937f51d23c0b12e0057cba223dfcd4"} Sep 09 02:40:58.487107 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:40:58.487060 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-758d56c575-xr2ml" podStartSLOduration=1.844939219 podStartE2EDuration="2.487044676s" totalImagesPullingTime="642.105457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:40:58.485838811 +0000 UTC m=+646.616080905" watchObservedRunningTime="2026-09-09 02:40:58.487044676 +0000 UTC m=+646.617286771" Sep 09 02:41:03.819297 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:03.819250 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:03.919232 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:03.919196 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:03.998121 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:03.998088 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:04.055648 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:04.055610 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7849498579-wsxch"] Sep 09 02:41:04.074011 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:04.073936 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7849498579-wsxch" Sep 09 02:41:04.076424 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:04.076371 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7849498579-wsxch"] Sep 09 02:41:04.077171 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:04.077146 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zlr72\"" Sep 09 02:41:04.078776 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:04.078283 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 09 02:41:04.078776 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:04.078424 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 09 02:41:04.078776 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:04.078508 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 09 02:41:04.080652 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:04.080626 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:04.187105 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:04.187078 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:04.214239 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:04.214204 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-859s8\" (UniqueName: \"kubernetes.io/projected/98f3f0ad-ca4f-4629-bfc0-2ab6aa722003-kube-api-access-859s8\") pod \"maas-controller-7849498579-wsxch\" (UID: \"98f3f0ad-ca4f-4629-bfc0-2ab6aa722003\") " pod="opendatahub/maas-controller-7849498579-wsxch" Sep 09 02:41:04.214239 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:04.214242 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/98f3f0ad-ca4f-4629-bfc0-2ab6aa722003-webhook-cert\") pod \"maas-controller-7849498579-wsxch\" (UID: \"98f3f0ad-ca4f-4629-bfc0-2ab6aa722003\") " pod="opendatahub/maas-controller-7849498579-wsxch" Sep 09 02:41:04.214442 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:04.214335 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98f3f0ad-ca4f-4629-bfc0-2ab6aa722003-metrics-certs\") pod \"maas-controller-7849498579-wsxch\" (UID: \"98f3f0ad-ca4f-4629-bfc0-2ab6aa722003\") " pod="opendatahub/maas-controller-7849498579-wsxch" Sep 09 02:41:04.315108 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:04.315078 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98f3f0ad-ca4f-4629-bfc0-2ab6aa722003-metrics-certs\") pod \"maas-controller-7849498579-wsxch\" (UID: \"98f3f0ad-ca4f-4629-bfc0-2ab6aa722003\") " pod="opendatahub/maas-controller-7849498579-wsxch" Sep 09 02:41:04.315262 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:04.315145 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-859s8\" (UniqueName: \"kubernetes.io/projected/98f3f0ad-ca4f-4629-bfc0-2ab6aa722003-kube-api-access-859s8\") pod \"maas-controller-7849498579-wsxch\" (UID: \"98f3f0ad-ca4f-4629-bfc0-2ab6aa722003\") " pod="opendatahub/maas-controller-7849498579-wsxch" Sep 09 02:41:04.315262 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:04.315165 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/98f3f0ad-ca4f-4629-bfc0-2ab6aa722003-webhook-cert\") pod \"maas-controller-7849498579-wsxch\" (UID: \"98f3f0ad-ca4f-4629-bfc0-2ab6aa722003\") " pod="opendatahub/maas-controller-7849498579-wsxch" Sep 09 02:41:04.317521 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:04.317497 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98f3f0ad-ca4f-4629-bfc0-2ab6aa722003-metrics-certs\") pod \"maas-controller-7849498579-wsxch\" (UID: \"98f3f0ad-ca4f-4629-bfc0-2ab6aa722003\") " pod="opendatahub/maas-controller-7849498579-wsxch" Sep 09 02:41:04.317646 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:04.317539 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/98f3f0ad-ca4f-4629-bfc0-2ab6aa722003-webhook-cert\") pod \"maas-controller-7849498579-wsxch\" (UID: \"98f3f0ad-ca4f-4629-bfc0-2ab6aa722003\") " pod="opendatahub/maas-controller-7849498579-wsxch" Sep 09 02:41:04.324396 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:04.324328 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-859s8\" (UniqueName: \"kubernetes.io/projected/98f3f0ad-ca4f-4629-bfc0-2ab6aa722003-kube-api-access-859s8\") pod \"maas-controller-7849498579-wsxch\" (UID: \"98f3f0ad-ca4f-4629-bfc0-2ab6aa722003\") " pod="opendatahub/maas-controller-7849498579-wsxch" Sep 09 02:41:04.386986 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:04.386957 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7849498579-wsxch" Sep 09 02:41:04.662423 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:04.662396 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7849498579-wsxch"] Sep 09 02:41:04.664021 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:41:04.663991 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f3f0ad_ca4f_4629_bfc0_2ab6aa722003.slice/crio-66792ff07a8b8a71a7193ea4dd05ca0d1022fe6f5affb26fed77f52911db49e7 WatchSource:0}: Error finding container 66792ff07a8b8a71a7193ea4dd05ca0d1022fe6f5affb26fed77f52911db49e7: Status 404 returned error can't find the container with id 66792ff07a8b8a71a7193ea4dd05ca0d1022fe6f5affb26fed77f52911db49e7 Sep 09 02:41:05.506206 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:05.506170 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7849498579-wsxch" event={"ID":"98f3f0ad-ca4f-4629-bfc0-2ab6aa722003","Type":"ContainerStarted","Data":"66792ff07a8b8a71a7193ea4dd05ca0d1022fe6f5affb26fed77f52911db49e7"} Sep 09 02:41:06.973176 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:06.973142 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:07.078041 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:07.077950 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:07.518513 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:07.518481 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7849498579-wsxch" event={"ID":"98f3f0ad-ca4f-4629-bfc0-2ab6aa722003","Type":"ContainerStarted","Data":"b3f58ed68d21bf1d6db46f80b39c2af1793cc8dd6e8d3d7ae03acdfbebb8a01c"} Sep 09 02:41:07.518689 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:07.518555 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7849498579-wsxch" Sep 09 02:41:07.554898 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:07.554841 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7849498579-wsxch" podStartSLOduration=1.495458137 podStartE2EDuration="3.554812063s" totalImagesPullingTime="2.059353926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:41:07.552560002 +0000 UTC m=+655.682802124" watchObservedRunningTime="2026-09-09 02:41:07.554812063 +0000 UTC m=+655.685054158" Sep 09 02:41:18.525942 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:18.525900 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7849498579-wsxch" Sep 09 02:41:23.679462 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:23.679424 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:23.746522 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:23.746492 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:26.357561 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:26.357522 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:26.459987 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:26.459955 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:38.569217 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:38.569183 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:38.664324 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:38.664278 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:40.158341 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.158311 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:40.253228 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.253186 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:40.280169 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.280137 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-55c9b5658-6wg48"] Sep 09 02:41:40.286123 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.286106 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" Sep 09 02:41:40.288780 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.288759 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 09 02:41:40.289891 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.289868 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hq897\"" Sep 09 02:41:40.290010 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.289890 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 09 02:41:40.295886 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.295861 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55c9b5658-6wg48"] Sep 09 02:41:40.341401 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.341370 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-openshift-service-ca\") pod \"maas-api-55c9b5658-6wg48\" (UID: \"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" Sep 09 02:41:40.341608 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.341437 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfns6\" (UniqueName: \"kubernetes.io/projected/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-kube-api-access-nfns6\") pod \"maas-api-55c9b5658-6wg48\" (UID: \"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" Sep 09 02:41:40.341608 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.341507 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-metrics-certs\") pod \"maas-api-55c9b5658-6wg48\" (UID: \"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" Sep 09 02:41:40.341608 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.341533 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-maas-api-tls\") pod \"maas-api-55c9b5658-6wg48\" (UID: \"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" Sep 09 02:41:40.442325 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.442234 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-maas-api-tls\") pod \"maas-api-55c9b5658-6wg48\" (UID: \"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" Sep 09 02:41:40.442325 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.442298 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-openshift-service-ca\") pod \"maas-api-55c9b5658-6wg48\" (UID: \"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" Sep 09 02:41:40.442639 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.442338 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfns6\" (UniqueName: \"kubernetes.io/projected/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-kube-api-access-nfns6\") pod \"maas-api-55c9b5658-6wg48\" (UID: \"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" Sep 09 02:41:40.442639 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.442403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-metrics-certs\") pod \"maas-api-55c9b5658-6wg48\" (UID: \"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" Sep 09 02:41:40.442639 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:41:40.442559 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found Sep 09 02:41:40.442639 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:41:40.442626 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-metrics-certs podName:8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4 nodeName:}" failed. No retries permitted until 2026-09-09 02:41:40.942607356 +0000 UTC m=+689.072849445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-metrics-certs") pod "maas-api-55c9b5658-6wg48" (UID: "8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4") : secret "maas-api-metrics-tls" not found Sep 09 02:41:40.443243 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.443220 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-openshift-service-ca\") pod \"maas-api-55c9b5658-6wg48\" (UID: \"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" Sep 09 02:41:40.445224 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.445203 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-maas-api-tls\") pod \"maas-api-55c9b5658-6wg48\" (UID: \"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" Sep 09 02:41:40.454141 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.454116 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfns6\" (UniqueName: \"kubernetes.io/projected/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-kube-api-access-nfns6\") pod \"maas-api-55c9b5658-6wg48\" (UID: \"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" Sep 09 02:41:40.605989 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.605957 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:40.712030 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.711944 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:40.816867 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.816836 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:40.913349 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.913298 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-rggjg"] Sep 09 02:41:40.917696 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.917675 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rggjg" Sep 09 02:41:40.920561 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.920537 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-pfmqc\"" Sep 09 02:41:40.920864 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.920848 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 09 02:41:40.933217 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.933188 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-rggjg"] Sep 09 02:41:40.947065 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.947036 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-metrics-certs\") pod \"maas-api-55c9b5658-6wg48\" (UID: \"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" Sep 09 02:41:40.949433 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:40.949401 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-metrics-certs\") pod \"maas-api-55c9b5658-6wg48\" (UID: \"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" Sep 09 02:41:41.047854 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:41.047755 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd8d2bf-69f5-4f60-ae12-ff382fa47f7e-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-rggjg\" (UID: \"7bd8d2bf-69f5-4f60-ae12-ff382fa47f7e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rggjg" Sep 09 02:41:41.048012 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:41.047893 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgjdd\" (UniqueName: \"kubernetes.io/projected/7bd8d2bf-69f5-4f60-ae12-ff382fa47f7e-kube-api-access-vgjdd\") pod \"payload-pre-processing-59fbc47bf8-rggjg\" (UID: \"7bd8d2bf-69f5-4f60-ae12-ff382fa47f7e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rggjg" Sep 09 02:41:41.149213 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:41.149175 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd8d2bf-69f5-4f60-ae12-ff382fa47f7e-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-rggjg\" (UID: \"7bd8d2bf-69f5-4f60-ae12-ff382fa47f7e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rggjg" Sep 09 02:41:41.149420 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:41.149289 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgjdd\" (UniqueName: \"kubernetes.io/projected/7bd8d2bf-69f5-4f60-ae12-ff382fa47f7e-kube-api-access-vgjdd\") pod \"payload-pre-processing-59fbc47bf8-rggjg\" (UID: \"7bd8d2bf-69f5-4f60-ae12-ff382fa47f7e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rggjg" Sep 09 02:41:41.150015 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:41.149989 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd8d2bf-69f5-4f60-ae12-ff382fa47f7e-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-rggjg\" (UID: \"7bd8d2bf-69f5-4f60-ae12-ff382fa47f7e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rggjg" Sep 09 02:41:41.160583 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:41.160546 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgjdd\" (UniqueName: \"kubernetes.io/projected/7bd8d2bf-69f5-4f60-ae12-ff382fa47f7e-kube-api-access-vgjdd\") pod \"payload-pre-processing-59fbc47bf8-rggjg\" (UID: \"7bd8d2bf-69f5-4f60-ae12-ff382fa47f7e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rggjg" Sep 09 02:41:41.198654 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:41.198603 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" Sep 09 02:41:41.235883 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:41.235851 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rggjg" Sep 09 02:41:41.485286 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:41.485258 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55c9b5658-6wg48"] Sep 09 02:41:41.487428 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:41:41.487398 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c42873d_b0e9_43a9_aac3_ebca4ff9e1a4.slice/crio-7ea8af1b48b02042174bc1fc706a810f22217d2278b9c09e4cd80917c67cee85 WatchSource:0}: Error finding container 7ea8af1b48b02042174bc1fc706a810f22217d2278b9c09e4cd80917c67cee85: Status 404 returned error can't find the container with id 7ea8af1b48b02042174bc1fc706a810f22217d2278b9c09e4cd80917c67cee85 Sep 09 02:41:41.489159 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:41.489141 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 02:41:41.534745 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:41.534709 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-rggjg"] Sep 09 02:41:41.538331 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:41:41.538303 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd8d2bf_69f5_4f60_ae12_ff382fa47f7e.slice/crio-aebec3b62c97a536434b1cd5b20d80c3264d1af7e07e319df001a3e25ed86c5a WatchSource:0}: Error finding container aebec3b62c97a536434b1cd5b20d80c3264d1af7e07e319df001a3e25ed86c5a: Status 404 returned error can't find the container with id aebec3b62c97a536434b1cd5b20d80c3264d1af7e07e319df001a3e25ed86c5a Sep 09 02:41:41.688744 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:41.688714 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rggjg" event={"ID":"7bd8d2bf-69f5-4f60-ae12-ff382fa47f7e","Type":"ContainerStarted","Data":"aebec3b62c97a536434b1cd5b20d80c3264d1af7e07e319df001a3e25ed86c5a"} Sep 09 02:41:41.689924 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:41.689896 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" event={"ID":"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4","Type":"ContainerStarted","Data":"7ea8af1b48b02042174bc1fc706a810f22217d2278b9c09e4cd80917c67cee85"} Sep 09 02:41:42.791160 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:42.791126 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:43.702495 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:43.702456 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rggjg" event={"ID":"7bd8d2bf-69f5-4f60-ae12-ff382fa47f7e","Type":"ContainerStarted","Data":"2431b4cb14698cbaa9fc3aed41b4859833822f63711e5f3f933a75af6e999299"} Sep 09 02:41:43.702684 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:43.702584 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rggjg" Sep 09 02:41:43.704055 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:43.704028 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" event={"ID":"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4","Type":"ContainerStarted","Data":"8e2ee62d33c8e89811797994554c72c4675cb4058f2dfb2ac88d85c6602cdc30"} Sep 09 02:41:43.704174 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:43.704076 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" Sep 09 02:41:43.722697 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:43.722651 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rggjg" podStartSLOduration=1.6968288409999999 podStartE2EDuration="3.722633088s" totalImagesPullingTime="2.025804247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:41:43.720360967 +0000 UTC m=+691.850603062" watchObservedRunningTime="2026-09-09 02:41:43.722633088 +0000 UTC m=+691.852875182" Sep 09 02:41:43.741965 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:43.741920 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" podStartSLOduration=1.665286322 podStartE2EDuration="3.74190461s" totalImagesPullingTime="2.076618288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:41:43.74017531 +0000 UTC m=+691.870417407" watchObservedRunningTime="2026-09-09 02:41:43.74190461 +0000 UTC m=+691.872146705" Sep 09 02:41:47.987184 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:47.987143 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:48.088172 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:48.088133 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:49.714447 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:49.714418 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" Sep 09 02:41:50.717025 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:50.716986 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:50.814131 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:50.814103 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:41:54.710750 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:41:54.710720 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rggjg" Sep 09 02:42:02.458880 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.458844 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl"] Sep 09 02:42:02.463539 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.463514 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:02.466426 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.466399 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-t6s5c\"" Sep 09 02:42:02.466566 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.466433 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 09 02:42:02.466566 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.466524 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 09 02:42:02.467693 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.467675 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 09 02:42:02.473091 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.473063 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl"] Sep 09 02:42:02.490493 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.490461 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:42:02.552871 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.552838 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78249e2b-fb45-4687-bf00-22c6e47645ef-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl\" (UID: \"78249e2b-fb45-4687-bf00-22c6e47645ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:02.553058 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.552886 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78249e2b-fb45-4687-bf00-22c6e47645ef-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl\" (UID: \"78249e2b-fb45-4687-bf00-22c6e47645ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:02.553058 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.552936 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnbcn\" (UniqueName: \"kubernetes.io/projected/78249e2b-fb45-4687-bf00-22c6e47645ef-kube-api-access-qnbcn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl\" (UID: \"78249e2b-fb45-4687-bf00-22c6e47645ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:02.553058 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.553006 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78249e2b-fb45-4687-bf00-22c6e47645ef-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl\" (UID: \"78249e2b-fb45-4687-bf00-22c6e47645ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:02.553191 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.553075 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78249e2b-fb45-4687-bf00-22c6e47645ef-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl\" (UID: \"78249e2b-fb45-4687-bf00-22c6e47645ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:02.553191 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.553141 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78249e2b-fb45-4687-bf00-22c6e47645ef-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl\" (UID: \"78249e2b-fb45-4687-bf00-22c6e47645ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:02.653786 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.653749 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78249e2b-fb45-4687-bf00-22c6e47645ef-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl\" (UID: \"78249e2b-fb45-4687-bf00-22c6e47645ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:02.653976 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.653835 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78249e2b-fb45-4687-bf00-22c6e47645ef-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl\" (UID: \"78249e2b-fb45-4687-bf00-22c6e47645ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:02.653976 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.653872 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78249e2b-fb45-4687-bf00-22c6e47645ef-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl\" (UID: \"78249e2b-fb45-4687-bf00-22c6e47645ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:02.653976 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.653902 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnbcn\" (UniqueName: \"kubernetes.io/projected/78249e2b-fb45-4687-bf00-22c6e47645ef-kube-api-access-qnbcn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl\" (UID: \"78249e2b-fb45-4687-bf00-22c6e47645ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:02.653976 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.653947 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78249e2b-fb45-4687-bf00-22c6e47645ef-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl\" (UID: \"78249e2b-fb45-4687-bf00-22c6e47645ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:02.654272 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.653990 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78249e2b-fb45-4687-bf00-22c6e47645ef-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl\" (UID: \"78249e2b-fb45-4687-bf00-22c6e47645ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:02.654272 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.654256 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78249e2b-fb45-4687-bf00-22c6e47645ef-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl\" (UID: \"78249e2b-fb45-4687-bf00-22c6e47645ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:02.654388 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.654333 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78249e2b-fb45-4687-bf00-22c6e47645ef-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl\" (UID: \"78249e2b-fb45-4687-bf00-22c6e47645ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:02.654388 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.654328 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78249e2b-fb45-4687-bf00-22c6e47645ef-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl\" (UID: \"78249e2b-fb45-4687-bf00-22c6e47645ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:02.656147 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.656119 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78249e2b-fb45-4687-bf00-22c6e47645ef-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl\" (UID: \"78249e2b-fb45-4687-bf00-22c6e47645ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:02.656398 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.656380 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78249e2b-fb45-4687-bf00-22c6e47645ef-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl\" (UID: \"78249e2b-fb45-4687-bf00-22c6e47645ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:02.663374 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.663352 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnbcn\" (UniqueName: \"kubernetes.io/projected/78249e2b-fb45-4687-bf00-22c6e47645ef-kube-api-access-qnbcn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl\" (UID: \"78249e2b-fb45-4687-bf00-22c6e47645ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:02.776584 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:02.776490 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:03.081196 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:03.081148 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl"] Sep 09 02:42:03.084197 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:42:03.084164 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78249e2b_fb45_4687_bf00_22c6e47645ef.slice/crio-6bfa8dad39ddee1c2150ef8d28fde9e612945577838717226d204318cc8bd3fa WatchSource:0}: Error finding container 6bfa8dad39ddee1c2150ef8d28fde9e612945577838717226d204318cc8bd3fa: Status 404 returned error can't find the container with id 6bfa8dad39ddee1c2150ef8d28fde9e612945577838717226d204318cc8bd3fa Sep 09 02:42:03.810710 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:03.810664 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" event={"ID":"78249e2b-fb45-4687-bf00-22c6e47645ef","Type":"ContainerStarted","Data":"6bfa8dad39ddee1c2150ef8d28fde9e612945577838717226d204318cc8bd3fa"} Sep 09 02:42:04.817643 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:04.817610 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" event={"ID":"78249e2b-fb45-4687-bf00-22c6e47645ef","Type":"ContainerStarted","Data":"19f20821044b0a916b4350401943ca7fc611481d5fac30973f45e43cd37d6e5c"} Sep 09 02:42:04.836914 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:04.836848 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" podStartSLOduration=1.6214051729999999 podStartE2EDuration="2.836804456s" totalImagesPullingTime="1.215399283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:42:04.836352149 +0000 UTC m=+712.966594248" watchObservedRunningTime="2026-09-09 02:42:04.836804456 +0000 UTC m=+712.967046553" Sep 09 02:42:12.489076 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:12.489041 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:42:12.777689 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:12.777609 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:12.777689 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:12.777672 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:12.790281 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:12.790256 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:12.868023 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:12.867998 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl" Sep 09 02:42:28.054960 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:28.054919 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:42:36.936882 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:36.936848 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:42:46.783012 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:46.782979 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:42:49.390725 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:49.390694 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:42:49.491626 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:49.491588 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:42:54.176891 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:42:54.176853 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:43:16.296660 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:43:16.296624 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:44:41.246399 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:41.246366 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:44:44.983204 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:44.983167 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-758d56c575-xr2ml"] Sep 09 02:44:44.983692 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:44.983429 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-758d56c575-xr2ml" podUID="a58cef93-7d4f-4149-aa55-66d78624cf75" containerName="authorino" containerID="cri-o://ad99b666e1c52c380d04294ae8f615b6ba937f51d23c0b12e0057cba223dfcd4" gracePeriod=30 Sep 09 02:44:45.249374 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:45.249347 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-758d56c575-xr2ml" Sep 09 02:44:45.373218 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:45.373184 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a58cef93-7d4f-4149-aa55-66d78624cf75-tls-cert\") pod \"a58cef93-7d4f-4149-aa55-66d78624cf75\" (UID: \"a58cef93-7d4f-4149-aa55-66d78624cf75\") " Sep 09 02:44:45.373401 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:45.373228 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bc5n\" (UniqueName: \"kubernetes.io/projected/a58cef93-7d4f-4149-aa55-66d78624cf75-kube-api-access-8bc5n\") pod \"a58cef93-7d4f-4149-aa55-66d78624cf75\" (UID: \"a58cef93-7d4f-4149-aa55-66d78624cf75\") " Sep 09 02:44:45.375322 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:45.375296 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58cef93-7d4f-4149-aa55-66d78624cf75-kube-api-access-8bc5n" (OuterVolumeSpecName: "kube-api-access-8bc5n") pod "a58cef93-7d4f-4149-aa55-66d78624cf75" (UID: "a58cef93-7d4f-4149-aa55-66d78624cf75"). InnerVolumeSpecName "kube-api-access-8bc5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:44:45.385246 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:45.385221 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a58cef93-7d4f-4149-aa55-66d78624cf75-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a58cef93-7d4f-4149-aa55-66d78624cf75" (UID: "a58cef93-7d4f-4149-aa55-66d78624cf75"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:44:45.474349 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:45.474319 2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a58cef93-7d4f-4149-aa55-66d78624cf75-tls-cert\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:44:45.474349 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:45.474346 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bc5n\" (UniqueName: \"kubernetes.io/projected/a58cef93-7d4f-4149-aa55-66d78624cf75-kube-api-access-8bc5n\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:44:45.527370 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:45.527281 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:44:45.600913 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:45.600882 2577 generic.go:356] "Generic (PLEG): container finished" podID="a58cef93-7d4f-4149-aa55-66d78624cf75" containerID="ad99b666e1c52c380d04294ae8f615b6ba937f51d23c0b12e0057cba223dfcd4" exitCode=0 Sep 09 02:44:45.601084 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:45.600924 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-758d56c575-xr2ml" Sep 09 02:44:45.601084 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:45.600940 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-758d56c575-xr2ml" event={"ID":"a58cef93-7d4f-4149-aa55-66d78624cf75","Type":"ContainerDied","Data":"ad99b666e1c52c380d04294ae8f615b6ba937f51d23c0b12e0057cba223dfcd4"} Sep 09 02:44:45.601084 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:45.600974 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-758d56c575-xr2ml" event={"ID":"a58cef93-7d4f-4149-aa55-66d78624cf75","Type":"ContainerDied","Data":"a2c9ae7db540b80ad846084708b1742f4284f4ef9e7c4be59bb8f797f373a2cc"} Sep 09 02:44:45.601084 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:45.600992 2577 scope.go:117] "RemoveContainer" containerID="ad99b666e1c52c380d04294ae8f615b6ba937f51d23c0b12e0057cba223dfcd4" Sep 09 02:44:45.609936 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:45.609916 2577 scope.go:117] "RemoveContainer" containerID="ad99b666e1c52c380d04294ae8f615b6ba937f51d23c0b12e0057cba223dfcd4" Sep 09 02:44:45.610200 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:44:45.610178 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad99b666e1c52c380d04294ae8f615b6ba937f51d23c0b12e0057cba223dfcd4\": container with ID starting with ad99b666e1c52c380d04294ae8f615b6ba937f51d23c0b12e0057cba223dfcd4 not found: ID does not exist" containerID="ad99b666e1c52c380d04294ae8f615b6ba937f51d23c0b12e0057cba223dfcd4" Sep 09 02:44:45.610273 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:45.610209 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad99b666e1c52c380d04294ae8f615b6ba937f51d23c0b12e0057cba223dfcd4"} err="failed to get container status \"ad99b666e1c52c380d04294ae8f615b6ba937f51d23c0b12e0057cba223dfcd4\": rpc error: code = NotFound desc = could not find container \"ad99b666e1c52c380d04294ae8f615b6ba937f51d23c0b12e0057cba223dfcd4\": container with ID starting with ad99b666e1c52c380d04294ae8f615b6ba937f51d23c0b12e0057cba223dfcd4 not found: ID does not exist" Sep 09 02:44:45.637561 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:45.637531 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-758d56c575-xr2ml"] Sep 09 02:44:45.641448 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:45.641426 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-758d56c575-xr2ml"] Sep 09 02:44:45.684142 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:45.684119 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:44:46.309476 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:46.309446 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a58cef93-7d4f-4149-aa55-66d78624cf75" path="/var/lib/kubelet/pods/a58cef93-7d4f-4149-aa55-66d78624cf75/volumes" Sep 09 02:44:46.654879 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:46.654832 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:44:46.753343 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:44:46.753265 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:03.110551 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.110498 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:03.358269 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.358224 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:03.393995 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.393959 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8"] Sep 09 02:45:03.394470 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.394450 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58cef93-7d4f-4149-aa55-66d78624cf75" containerName="authorino" Sep 09 02:45:03.394592 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.394473 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58cef93-7d4f-4149-aa55-66d78624cf75" containerName="authorino" Sep 09 02:45:03.394657 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.394617 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a58cef93-7d4f-4149-aa55-66d78624cf75" containerName="authorino" Sep 09 02:45:03.399144 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.399126 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.401863 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.401838 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 09 02:45:03.401863 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.401854 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b18109-openshift-default-dockercfg-4jkds\"" Sep 09 02:45:03.409747 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.409718 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8"] Sep 09 02:45:03.531395 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.531359 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee01b31-2432-4494-a93c-20cd8e73fefa-istiod-ca-cert\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.531395 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.531398 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-credential-socket\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.531620 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.531423 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-workload-certs\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.531620 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.531526 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-token\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.531620 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.531556 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd6sz\" (UniqueName: \"kubernetes.io/projected/aee01b31-2432-4494-a93c-20cd8e73fefa-kube-api-access-kd6sz\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.531620 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.531584 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-podinfo\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.531774 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.531668 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-envoy\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.531774 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.531692 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-workload-socket\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.531774 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.531724 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-data\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.595796 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.595761 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:03.633120 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.633087 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-envoy\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.633268 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.633132 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-workload-socket\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.633268 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.633169 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-data\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.633268 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.633215 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee01b31-2432-4494-a93c-20cd8e73fefa-istiod-ca-cert\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.633268 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.633240 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-credential-socket\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.633268 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.633261 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-workload-certs\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.633526 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.633320 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-token\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.633526 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.633344 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd6sz\" (UniqueName: \"kubernetes.io/projected/aee01b31-2432-4494-a93c-20cd8e73fefa-kube-api-access-kd6sz\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.633526 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.633386 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-podinfo\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.633673 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.633638 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-data\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.633752 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.633724 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-workload-certs\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.633834 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.633771 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-workload-socket\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.634009 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.633988 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-credential-socket\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.634095 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.634035 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee01b31-2432-4494-a93c-20cd8e73fefa-istiod-ca-cert\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.635655 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.635634 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-envoy\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.635896 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.635877 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-podinfo\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.642401 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.642379 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-token\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.642484 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.642408 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd6sz\" (UniqueName: \"kubernetes.io/projected/aee01b31-2432-4494-a93c-20cd8e73fefa-kube-api-access-kd6sz\") pod \"e2e-inf-a-b18109-openshift-default-758cc57864-s86q8\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.736110 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.736037 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:03.802034 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:03.801884 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:04.015554 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:04.015531 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8"] Sep 09 02:45:04.017583 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:45:04.017553 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee01b31_2432_4494_a93c_20cd8e73fefa.slice/crio-976506c7f09aceee3a4ee3e82b66ade30d920b784151a05608e2005255dc1946 WatchSource:0}: Error finding container 976506c7f09aceee3a4ee3e82b66ade30d920b784151a05608e2005255dc1946: Status 404 returned error can't find the container with id 976506c7f09aceee3a4ee3e82b66ade30d920b784151a05608e2005255dc1946 Sep 09 02:45:04.549270 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:04.549239 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:04.637768 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:04.637739 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:04.698002 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:04.697963 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" event={"ID":"aee01b31-2432-4494-a93c-20cd8e73fefa","Type":"ContainerStarted","Data":"976506c7f09aceee3a4ee3e82b66ade30d920b784151a05608e2005255dc1946"} Sep 09 02:45:04.989606 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:04.989533 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:04.994967 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:04.994943 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7879-140597200113072"] Sep 09 02:45:04.999363 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:04.999348 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-140597200113072" Sep 09 02:45:05.002327 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:05.002305 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gnkf7\"" Sep 09 02:45:05.004432 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:05.004411 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7879-140597200113072"] Sep 09 02:45:05.045630 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:05.045597 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kjvp\" (UniqueName: \"kubernetes.io/projected/e95b2f7d-5b10-4c12-806a-1e2242a128f6-kube-api-access-7kjvp\") pod \"test-curl-7879-140597200113072\" (UID: \"e95b2f7d-5b10-4c12-806a-1e2242a128f6\") " pod="opendatahub/test-curl-7879-140597200113072" Sep 09 02:45:05.146638 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:05.146606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kjvp\" (UniqueName: \"kubernetes.io/projected/e95b2f7d-5b10-4c12-806a-1e2242a128f6-kube-api-access-7kjvp\") pod \"test-curl-7879-140597200113072\" (UID: \"e95b2f7d-5b10-4c12-806a-1e2242a128f6\") " pod="opendatahub/test-curl-7879-140597200113072" Sep 09 02:45:05.156096 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:05.156069 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kjvp\" (UniqueName: \"kubernetes.io/projected/e95b2f7d-5b10-4c12-806a-1e2242a128f6-kube-api-access-7kjvp\") pod \"test-curl-7879-140597200113072\" (UID: \"e95b2f7d-5b10-4c12-806a-1e2242a128f6\") " pod="opendatahub/test-curl-7879-140597200113072" Sep 09 02:45:05.310702 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:05.310617 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-140597200113072" Sep 09 02:45:05.581832 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:05.581786 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7879-140597200113072"] Sep 09 02:45:05.583802 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:45:05.583759 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode95b2f7d_5b10_4c12_806a_1e2242a128f6.slice/crio-e8d3069430d12fc3e87333994a1ac4b9d4f314d757a6595e31652ba940fd96bf WatchSource:0}: Error finding container e8d3069430d12fc3e87333994a1ac4b9d4f314d757a6595e31652ba940fd96bf: Status 404 returned error can't find the container with id e8d3069430d12fc3e87333994a1ac4b9d4f314d757a6595e31652ba940fd96bf Sep 09 02:45:05.705306 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:05.705268 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7879-140597200113072" event={"ID":"e95b2f7d-5b10-4c12-806a-1e2242a128f6","Type":"ContainerStarted","Data":"e8d3069430d12fc3e87333994a1ac4b9d4f314d757a6595e31652ba940fd96bf"} Sep 09 02:45:06.711936 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:06.711895 2577 generic.go:356] "Generic (PLEG): container finished" podID="e95b2f7d-5b10-4c12-806a-1e2242a128f6" containerID="735979b6abb5dcd1e4dd4145b22ff0c24639ff12a28015ab99b36800c2c73265" exitCode=0 Sep 09 02:45:06.712419 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:06.711963 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7879-140597200113072" event={"ID":"e95b2f7d-5b10-4c12-806a-1e2242a128f6","Type":"ContainerDied","Data":"735979b6abb5dcd1e4dd4145b22ff0c24639ff12a28015ab99b36800c2c73265"} Sep 09 02:45:08.040345 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.040322 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-140597200113072" Sep 09 02:45:08.062151 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.062121 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7879-140597200113072_e95b2f7d-5b10-4c12-806a-1e2242a128f6/test-curl-7879-140597200113072/0.log" Sep 09 02:45:08.087076 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.087046 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7879-140597200113072"] Sep 09 02:45:08.087730 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.087691 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:45:08.087887 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.087779 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:45:08.087887 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.087857 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:45:08.090235 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.090211 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7879-140597200113072"] Sep 09 02:45:08.176068 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.176034 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kjvp\" (UniqueName: \"kubernetes.io/projected/e95b2f7d-5b10-4c12-806a-1e2242a128f6-kube-api-access-7kjvp\") pod \"e95b2f7d-5b10-4c12-806a-1e2242a128f6\" (UID: \"e95b2f7d-5b10-4c12-806a-1e2242a128f6\") " Sep 09 02:45:08.178090 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.178060 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95b2f7d-5b10-4c12-806a-1e2242a128f6-kube-api-access-7kjvp" (OuterVolumeSpecName: "kube-api-access-7kjvp") pod "e95b2f7d-5b10-4c12-806a-1e2242a128f6" (UID: "e95b2f7d-5b10-4c12-806a-1e2242a128f6"). InnerVolumeSpecName "kube-api-access-7kjvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:45:08.214424 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.214397 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7879-140597200113072"] Sep 09 02:45:08.214857 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.214843 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e95b2f7d-5b10-4c12-806a-1e2242a128f6" containerName="test-curl-7879-140597200113072" Sep 09 02:45:08.214909 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.214858 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e95b2f7d-5b10-4c12-806a-1e2242a128f6" containerName="test-curl-7879-140597200113072" Sep 09 02:45:08.214957 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.214948 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e95b2f7d-5b10-4c12-806a-1e2242a128f6" containerName="test-curl-7879-140597200113072" Sep 09 02:45:08.218073 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.218058 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-140597200113072" Sep 09 02:45:08.239053 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.237244 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7879-140597200113072"] Sep 09 02:45:08.276967 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.276894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s8kb\" (UniqueName: \"kubernetes.io/projected/7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d-kube-api-access-5s8kb\") pod \"test-curl-7879-140597200113072\" (UID: \"7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d\") " pod="opendatahub/test-curl-7879-140597200113072" Sep 09 02:45:08.277102 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.277043 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kjvp\" (UniqueName: \"kubernetes.io/projected/e95b2f7d-5b10-4c12-806a-1e2242a128f6-kube-api-access-7kjvp\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:45:08.309467 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.309437 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e95b2f7d-5b10-4c12-806a-1e2242a128f6" path="/var/lib/kubelet/pods/e95b2f7d-5b10-4c12-806a-1e2242a128f6/volumes" Sep 09 02:45:08.377693 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.377653 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s8kb\" (UniqueName: \"kubernetes.io/projected/7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d-kube-api-access-5s8kb\") pod \"test-curl-7879-140597200113072\" (UID: \"7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d\") " pod="opendatahub/test-curl-7879-140597200113072" Sep 09 02:45:08.386402 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.386372 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s8kb\" (UniqueName: \"kubernetes.io/projected/7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d-kube-api-access-5s8kb\") pod \"test-curl-7879-140597200113072\" (UID: \"7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d\") " pod="opendatahub/test-curl-7879-140597200113072" Sep 09 02:45:08.542237 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.542153 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-140597200113072" Sep 09 02:45:08.725042 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.725015 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-140597200113072" Sep 09 02:45:08.725221 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.725021 2577 scope.go:117] "RemoveContainer" containerID="735979b6abb5dcd1e4dd4145b22ff0c24639ff12a28015ab99b36800c2c73265" Sep 09 02:45:08.726691 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.726657 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" event={"ID":"aee01b31-2432-4494-a93c-20cd8e73fefa","Type":"ContainerStarted","Data":"3809d0a7eac02e3f1c90d1ebe8a102e89da24878ebf2656746657e6df150c042"} Sep 09 02:45:08.736675 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.736661 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:08.749606 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.749568 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" podStartSLOduration=1.681846781 podStartE2EDuration="5.749556238s" totalImagesPullingTime="4.067709457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:45:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:45:08.747960729 +0000 UTC m=+896.878202825" watchObservedRunningTime="2026-09-09 02:45:08.749556238 +0000 UTC m=+896.879798420" Sep 09 02:45:08.814395 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:08.814367 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7879-140597200113072"] Sep 09 02:45:08.817021 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:45:08.816991 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b05fccb_6a8e_4740_8dd3_f251ce5c0b6d.slice/crio-73e7d174856570addfde689d44118183f939bac3cd1005b97e4aec9bcd1b934a WatchSource:0}: Error finding container 73e7d174856570addfde689d44118183f939bac3cd1005b97e4aec9bcd1b934a: Status 404 returned error can't find the container with id 73e7d174856570addfde689d44118183f939bac3cd1005b97e4aec9bcd1b934a Sep 09 02:45:09.733706 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:09.733619 2577 generic.go:356] "Generic (PLEG): container finished" podID="7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d" containerID="ec6c0ca43a66a67af5d919cbd86c59156fa24e7a7d609d2c84c473e34ece5067" exitCode=0 Sep 09 02:45:09.734101 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:09.733706 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7879-140597200113072" event={"ID":"7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d","Type":"ContainerDied","Data":"ec6c0ca43a66a67af5d919cbd86c59156fa24e7a7d609d2c84c473e34ece5067"} Sep 09 02:45:09.734101 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:09.733746 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7879-140597200113072" event={"ID":"7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d","Type":"ContainerStarted","Data":"73e7d174856570addfde689d44118183f939bac3cd1005b97e4aec9bcd1b934a"} Sep 09 02:45:09.741788 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:09.741762 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:10.646995 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:10.646963 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:10.737448 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:10.737413 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:10.739508 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:10.739469 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:10.740433 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:10.740415 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:10.873802 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:10.873777 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-140597200113072" Sep 09 02:45:10.895256 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:10.895233 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7879-140597200113072_7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d/test-curl-7879-140597200113072/0.log" Sep 09 02:45:10.919495 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:10.919415 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7879-140597200113072"] Sep 09 02:45:10.923381 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:10.923356 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7879-140597200113072"] Sep 09 02:45:11.001983 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.001952 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5s8kb\" (UniqueName: \"kubernetes.io/projected/7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d-kube-api-access-5s8kb\") pod \"7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d\" (UID: \"7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d\") " Sep 09 02:45:11.004028 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.004003 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d-kube-api-access-5s8kb" (OuterVolumeSpecName: "kube-api-access-5s8kb") pod "7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d" (UID: "7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d"). InnerVolumeSpecName "kube-api-access-5s8kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:45:11.102868 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.102803 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s8kb\" (UniqueName: \"kubernetes.io/projected/7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d-kube-api-access-5s8kb\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:45:11.399568 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.399535 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:11.512436 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.512405 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:11.613681 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.613646 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:11.715266 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.715182 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:11.745881 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.745854 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-140597200113072" Sep 09 02:45:11.746287 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.745885 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73e7d174856570addfde689d44118183f939bac3cd1005b97e4aec9bcd1b934a" Sep 09 02:45:11.755443 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:45:11.755410 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7879-140597200113072\" is forbidden: User \"system:node:ip-10-0-137-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-37.ec2.internal' and this object" podUID="7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d" pod="opendatahub/test-curl-7879-140597200113072" Sep 09 02:45:11.772682 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.772655 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj"] Sep 09 02:45:11.773166 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.773148 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d" containerName="test-curl-7879-140597200113072" Sep 09 02:45:11.773282 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.773169 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d" containerName="test-curl-7879-140597200113072" Sep 09 02:45:11.773282 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.773267 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d" containerName="test-curl-7879-140597200113072" Sep 09 02:45:11.776561 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.776540 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:11.779699 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.779681 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-07b2d4-openshift-default-dockercfg-n4xpr\"" Sep 09 02:45:11.793333 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.793303 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj"] Sep 09 02:45:11.816592 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:45:11.816565 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7879-140597200113072\" is forbidden: User \"system:node:ip-10-0-137-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-37.ec2.internal' and this object" podUID="7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d" pod="opendatahub/test-curl-7879-140597200113072" Sep 09 02:45:11.821093 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.821071 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2"] Sep 09 02:45:11.823607 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.823579 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:11.824500 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.824483 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" Sep 09 02:45:11.826961 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:45:11.826934 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7879-140597200113072\" is forbidden: User \"system:node:ip-10-0-137-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-37.ec2.internal' and this object" podUID="7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d" pod="opendatahub/test-curl-7879-140597200113072" Sep 09 02:45:11.827060 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.827001 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b18109-serving-cert\"" Sep 09 02:45:11.838446 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.838409 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2"] Sep 09 02:45:11.911460 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.911427 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7aebac-c330-497e-a0bd-ed44095a0486-metrics-certs\") pod \"maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2\" (UID: \"fb7aebac-c330-497e-a0bd-ed44095a0486\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" Sep 09 02:45:11.911697 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.911466 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb7aebac-c330-497e-a0bd-ed44095a0486-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2\" (UID: \"fb7aebac-c330-497e-a0bd-ed44095a0486\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" Sep 09 02:45:11.911697 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.911488 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cfmz\" (UniqueName: \"kubernetes.io/projected/0b5deaea-bded-4181-99aa-69d4a867daaa-kube-api-access-7cfmz\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:11.911697 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.911608 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-envoy\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:11.911697 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.911647 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-workload-socket\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:11.911892 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.911729 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-data\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:11.911892 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.911759 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-podinfo\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:11.911892 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.911779 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmfkw\" (UniqueName: \"kubernetes.io/projected/fb7aebac-c330-497e-a0bd-ed44095a0486-kube-api-access-zmfkw\") pod \"maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2\" (UID: \"fb7aebac-c330-497e-a0bd-ed44095a0486\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" Sep 09 02:45:11.911892 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.911797 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5deaea-bded-4181-99aa-69d4a867daaa-istiod-ca-cert\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:11.911892 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.911873 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-credential-socket\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:11.912070 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.911901 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb7aebac-c330-497e-a0bd-ed44095a0486-maas-api-tls\") pod \"maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2\" (UID: \"fb7aebac-c330-497e-a0bd-ed44095a0486\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" Sep 09 02:45:11.912070 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.911923 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-workload-certs\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:11.912070 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.911946 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-token\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:11.915159 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:11.915132 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:12.010846 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.010728 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:12.012899 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.012877 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7aebac-c330-497e-a0bd-ed44095a0486-metrics-certs\") pod \"maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2\" (UID: \"fb7aebac-c330-497e-a0bd-ed44095a0486\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" Sep 09 02:45:12.012997 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.012907 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb7aebac-c330-497e-a0bd-ed44095a0486-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2\" (UID: \"fb7aebac-c330-497e-a0bd-ed44095a0486\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" Sep 09 02:45:12.012997 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.012927 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cfmz\" (UniqueName: \"kubernetes.io/projected/0b5deaea-bded-4181-99aa-69d4a867daaa-kube-api-access-7cfmz\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:12.012997 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.012957 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-envoy\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:12.012997 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.012979 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-workload-socket\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:12.013216 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.013005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-data\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:12.013216 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.013021 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-podinfo\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:12.014183 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.013755 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-data\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:12.014183 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.013911 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmfkw\" (UniqueName: \"kubernetes.io/projected/fb7aebac-c330-497e-a0bd-ed44095a0486-kube-api-access-zmfkw\") pod \"maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2\" (UID: \"fb7aebac-c330-497e-a0bd-ed44095a0486\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" Sep 09 02:45:12.014183 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.013963 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5deaea-bded-4181-99aa-69d4a867daaa-istiod-ca-cert\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:12.014183 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.014067 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-workload-socket\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:12.015073 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.014268 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb7aebac-c330-497e-a0bd-ed44095a0486-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2\" (UID: \"fb7aebac-c330-497e-a0bd-ed44095a0486\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" Sep 09 02:45:12.015073 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.014297 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-credential-socket\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:12.015073 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.014368 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb7aebac-c330-497e-a0bd-ed44095a0486-maas-api-tls\") pod \"maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2\" (UID: \"fb7aebac-c330-497e-a0bd-ed44095a0486\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" Sep 09 02:45:12.015073 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.014418 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-workload-certs\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:12.015073 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.014453 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-token\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:12.015073 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.014637 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5deaea-bded-4181-99aa-69d4a867daaa-istiod-ca-cert\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:12.022546 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.015431 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-workload-certs\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:12.022546 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:45:12.015528 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-serving-cert: secret "maas-api-e2e-inf-a-b18109-serving-cert" not found Sep 09 02:45:12.022546 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:45:12.015666 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb7aebac-c330-497e-a0bd-ed44095a0486-maas-api-tls podName:fb7aebac-c330-497e-a0bd-ed44095a0486 nodeName:}" failed. No retries permitted until 2026-09-09 02:45:12.515637138 +0000 UTC m=+900.645879218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb7aebac-c330-497e-a0bd-ed44095a0486-maas-api-tls") pod "maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" (UID: "fb7aebac-c330-497e-a0bd-ed44095a0486") : secret "maas-api-e2e-inf-a-b18109-serving-cert" not found Sep 09 02:45:12.022546 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.016319 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-envoy\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:12.022546 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.016512 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-podinfo\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:12.022908 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.022616 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmfkw\" (UniqueName: \"kubernetes.io/projected/fb7aebac-c330-497e-a0bd-ed44095a0486-kube-api-access-zmfkw\") pod \"maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2\" (UID: \"fb7aebac-c330-497e-a0bd-ed44095a0486\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" Sep 09 02:45:12.022908 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.022761 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-credential-socket\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:12.023490 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.023044 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7aebac-c330-497e-a0bd-ed44095a0486-metrics-certs\") pod \"maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2\" (UID: \"fb7aebac-c330-497e-a0bd-ed44095a0486\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" Sep 09 02:45:12.025601 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.025575 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-token\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:12.026241 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.026220 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cfmz\" (UniqueName: \"kubernetes.io/projected/0b5deaea-bded-4181-99aa-69d4a867daaa-kube-api-access-7cfmz\") pod \"e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:12.086445 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.086412 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:12.087575 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.087549 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:12.186658 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.186616 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:12.309945 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:45:12.309858 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7879-140597200113072\" is forbidden: User \"system:node:ip-10-0-137-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-37.ec2.internal' and this object" podUID="7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d" pod="opendatahub/test-curl-7879-140597200113072" Sep 09 02:45:12.314800 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.314719 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d" path="/var/lib/kubelet/pods/7b05fccb-6a8e-4740-8dd3-f251ce5c0b6d/volumes" Sep 09 02:45:12.326787 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.326764 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/2.log" Sep 09 02:45:12.327354 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.327337 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/2.log" Sep 09 02:45:12.366228 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.366105 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:12.367523 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.367505 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj"] Sep 09 02:45:12.371458 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:45:12.371423 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5deaea_bded_4181_99aa_69d4a867daaa.slice/crio-82959b5aff22adaa4f35851abe07f075fdbd7b17bb034965ec264cbcdc0f4218 WatchSource:0}: Error finding container 82959b5aff22adaa4f35851abe07f075fdbd7b17bb034965ec264cbcdc0f4218: Status 404 returned error can't find the container with id 82959b5aff22adaa4f35851abe07f075fdbd7b17bb034965ec264cbcdc0f4218 Sep 09 02:45:12.373659 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.373629 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:45:12.373722 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.373691 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:45:12.373722 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.373718 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:45:12.510369 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.510342 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:12.520197 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.520167 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb7aebac-c330-497e-a0bd-ed44095a0486-maas-api-tls\") pod \"maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2\" (UID: \"fb7aebac-c330-497e-a0bd-ed44095a0486\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" Sep 09 02:45:12.522914 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.522876 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb7aebac-c330-497e-a0bd-ed44095a0486-maas-api-tls\") pod \"maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2\" (UID: \"fb7aebac-c330-497e-a0bd-ed44095a0486\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" Sep 09 02:45:12.617102 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.617024 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:12.738252 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.738226 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" Sep 09 02:45:12.753653 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.753614 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" event={"ID":"0b5deaea-bded-4181-99aa-69d4a867daaa","Type":"ContainerStarted","Data":"2e7d71b73df0a64a36efd486f29a2605e3f63c8ecb4ae9743e7b0b113db4bb3a"} Sep 09 02:45:12.754077 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.753682 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" event={"ID":"0b5deaea-bded-4181-99aa-69d4a867daaa","Type":"ContainerStarted","Data":"82959b5aff22adaa4f35851abe07f075fdbd7b17bb034965ec264cbcdc0f4218"} Sep 09 02:45:12.775929 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:12.775883 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" podStartSLOduration=1.775866119 podStartE2EDuration="1.775866119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:45:12.773565062 +0000 UTC m=+900.903807158" watchObservedRunningTime="2026-09-09 02:45:12.775866119 +0000 UTC m=+900.906108215" Sep 09 02:45:13.025495 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:13.025466 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2"] Sep 09 02:45:13.027690 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:45:13.027586 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7aebac_c330_497e_a0bd_ed44095a0486.slice/crio-ca7057216c2f3001654150356a73bd286feffe5d03c81a343496342ad05654f9 WatchSource:0}: Error finding container ca7057216c2f3001654150356a73bd286feffe5d03c81a343496342ad05654f9: Status 404 returned error can't find the container with id ca7057216c2f3001654150356a73bd286feffe5d03c81a343496342ad05654f9 Sep 09 02:45:13.087134 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:13.087100 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:13.088604 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:13.088577 2577 patch_prober.go:28] interesting pod/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 09 02:45:13.088701 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:13.088639 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" podUID="0b5deaea-bded-4181-99aa-69d4a867daaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 09 02:45:13.761484 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:13.761381 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" event={"ID":"fb7aebac-c330-497e-a0bd-ed44095a0486","Type":"ContainerStarted","Data":"815d6f5f03f501ae20a5675b25764ba4d9d31e78469c91fe9d210d516d2c587e"} Sep 09 02:45:13.761484 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:13.761451 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" event={"ID":"fb7aebac-c330-497e-a0bd-ed44095a0486","Type":"ContainerStarted","Data":"ca7057216c2f3001654150356a73bd286feffe5d03c81a343496342ad05654f9"} Sep 09 02:45:13.761950 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:13.761658 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" Sep 09 02:45:13.780351 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:13.780299 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" podStartSLOduration=2.332093965 podStartE2EDuration="2.780283518s" totalImagesPullingTime="448.189553ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:45:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:45:13.778078799 +0000 UTC m=+901.908320917" watchObservedRunningTime="2026-09-09 02:45:13.780283518 +0000 UTC m=+901.910525614" Sep 09 02:45:14.091111 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:14.091035 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:14.495494 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:14.495459 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:14.601310 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:14.601280 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:14.694310 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:14.694280 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:14.766885 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:14.766776 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:14.767522 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:14.767505 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:14.794575 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:14.794541 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:15.728055 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:15.728027 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95"] Sep 09 02:45:15.731826 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:15.731793 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" Sep 09 02:45:15.734755 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:15.734733 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-79a2702e-serving-cert\"" Sep 09 02:45:15.743924 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:15.743899 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95"] Sep 09 02:45:15.763687 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:15.763663 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:15.855726 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:15.855696 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5653c22-2a40-465a-888f-2aae14e21d39-maas-api-tls\") pod \"maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95\" (UID: \"c5653c22-2a40-465a-888f-2aae14e21d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" Sep 09 02:45:15.856195 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:15.855748 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdv9b\" (UniqueName: \"kubernetes.io/projected/c5653c22-2a40-465a-888f-2aae14e21d39-kube-api-access-sdv9b\") pod \"maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95\" (UID: \"c5653c22-2a40-465a-888f-2aae14e21d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" Sep 09 02:45:15.856195 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:15.855789 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5653c22-2a40-465a-888f-2aae14e21d39-metrics-certs\") pod \"maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95\" (UID: \"c5653c22-2a40-465a-888f-2aae14e21d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" Sep 09 02:45:15.856195 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:15.856053 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5653c22-2a40-465a-888f-2aae14e21d39-openshift-service-ca\") pod \"maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95\" (UID: \"c5653c22-2a40-465a-888f-2aae14e21d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" Sep 09 02:45:15.863931 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:15.863909 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:15.956755 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:15.956717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5653c22-2a40-465a-888f-2aae14e21d39-maas-api-tls\") pod \"maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95\" (UID: \"c5653c22-2a40-465a-888f-2aae14e21d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" Sep 09 02:45:15.956976 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:15.956776 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdv9b\" (UniqueName: \"kubernetes.io/projected/c5653c22-2a40-465a-888f-2aae14e21d39-kube-api-access-sdv9b\") pod \"maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95\" (UID: \"c5653c22-2a40-465a-888f-2aae14e21d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" Sep 09 02:45:15.956976 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:15.956793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5653c22-2a40-465a-888f-2aae14e21d39-metrics-certs\") pod \"maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95\" (UID: \"c5653c22-2a40-465a-888f-2aae14e21d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" Sep 09 02:45:15.956976 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:15.956844 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5653c22-2a40-465a-888f-2aae14e21d39-openshift-service-ca\") pod \"maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95\" (UID: \"c5653c22-2a40-465a-888f-2aae14e21d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" Sep 09 02:45:15.957489 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:15.957467 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5653c22-2a40-465a-888f-2aae14e21d39-openshift-service-ca\") pod \"maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95\" (UID: \"c5653c22-2a40-465a-888f-2aae14e21d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" Sep 09 02:45:15.959312 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:15.959290 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5653c22-2a40-465a-888f-2aae14e21d39-metrics-certs\") pod \"maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95\" (UID: \"c5653c22-2a40-465a-888f-2aae14e21d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" Sep 09 02:45:15.959407 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:15.959325 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5653c22-2a40-465a-888f-2aae14e21d39-maas-api-tls\") pod \"maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95\" (UID: \"c5653c22-2a40-465a-888f-2aae14e21d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" Sep 09 02:45:15.964721 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:15.964677 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:15.965365 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:15.965345 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdv9b\" (UniqueName: \"kubernetes.io/projected/c5653c22-2a40-465a-888f-2aae14e21d39-kube-api-access-sdv9b\") pod \"maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95\" (UID: \"c5653c22-2a40-465a-888f-2aae14e21d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" Sep 09 02:45:16.043937 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.043786 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" Sep 09 02:45:16.278430 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.278395 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:16.325725 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.325702 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95"] Sep 09 02:45:16.328938 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:45:16.328910 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5653c22_2a40_465a_888f_2aae14e21d39.slice/crio-51ee4ad33e7a76aad1ed42153f74acffb92bfdf0b7b71a43454671d81e024b02 WatchSource:0}: Error finding container 51ee4ad33e7a76aad1ed42153f74acffb92bfdf0b7b71a43454671d81e024b02: Status 404 returned error can't find the container with id 51ee4ad33e7a76aad1ed42153f74acffb92bfdf0b7b71a43454671d81e024b02 Sep 09 02:45:16.378366 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.378332 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:16.681692 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.681656 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4"] Sep 09 02:45:16.687112 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.687089 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" Sep 09 02:45:16.689930 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.689909 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-07b2d4-serving-cert\"" Sep 09 02:45:16.698314 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.698234 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4"] Sep 09 02:45:16.728965 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.728933 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:16.764482 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.764444 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fwnf\" (UniqueName: \"kubernetes.io/projected/99717fb1-9461-43dd-81bd-fc113435e140-kube-api-access-9fwnf\") pod \"maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4\" (UID: \"99717fb1-9461-43dd-81bd-fc113435e140\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" Sep 09 02:45:16.764640 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.764487 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99717fb1-9461-43dd-81bd-fc113435e140-maas-api-tls\") pod \"maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4\" (UID: \"99717fb1-9461-43dd-81bd-fc113435e140\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" Sep 09 02:45:16.764640 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.764569 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99717fb1-9461-43dd-81bd-fc113435e140-metrics-certs\") pod \"maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4\" (UID: \"99717fb1-9461-43dd-81bd-fc113435e140\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" Sep 09 02:45:16.764640 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.764618 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99717fb1-9461-43dd-81bd-fc113435e140-openshift-service-ca\") pod \"maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4\" (UID: \"99717fb1-9461-43dd-81bd-fc113435e140\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" Sep 09 02:45:16.778360 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.778321 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" event={"ID":"c5653c22-2a40-465a-888f-2aae14e21d39","Type":"ContainerStarted","Data":"823e3bcf67c42c5003ad2ba8c13a7ad86833e29d3a3322df5649cb4115199091"} Sep 09 02:45:16.778591 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.778401 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" Sep 09 02:45:16.778591 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.778427 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" event={"ID":"c5653c22-2a40-465a-888f-2aae14e21d39","Type":"ContainerStarted","Data":"51ee4ad33e7a76aad1ed42153f74acffb92bfdf0b7b71a43454671d81e024b02"} Sep 09 02:45:16.798704 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.798663 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" podStartSLOduration=1.46701323 podStartE2EDuration="1.798646234s" totalImagesPullingTime="331.633004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:45:16.797668282 +0000 UTC m=+904.927910379" watchObservedRunningTime="2026-09-09 02:45:16.798646234 +0000 UTC m=+904.928888327" Sep 09 02:45:16.807441 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.807399 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:16.865526 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.865498 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fwnf\" (UniqueName: \"kubernetes.io/projected/99717fb1-9461-43dd-81bd-fc113435e140-kube-api-access-9fwnf\") pod \"maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4\" (UID: \"99717fb1-9461-43dd-81bd-fc113435e140\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" Sep 09 02:45:16.865937 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.865535 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99717fb1-9461-43dd-81bd-fc113435e140-maas-api-tls\") pod \"maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4\" (UID: \"99717fb1-9461-43dd-81bd-fc113435e140\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" Sep 09 02:45:16.865937 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.865663 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99717fb1-9461-43dd-81bd-fc113435e140-metrics-certs\") pod \"maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4\" (UID: \"99717fb1-9461-43dd-81bd-fc113435e140\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" Sep 09 02:45:16.865937 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.865865 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99717fb1-9461-43dd-81bd-fc113435e140-openshift-service-ca\") pod \"maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4\" (UID: \"99717fb1-9461-43dd-81bd-fc113435e140\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" Sep 09 02:45:16.866655 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.866636 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99717fb1-9461-43dd-81bd-fc113435e140-openshift-service-ca\") pod \"maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4\" (UID: \"99717fb1-9461-43dd-81bd-fc113435e140\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" Sep 09 02:45:16.868120 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.868102 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99717fb1-9461-43dd-81bd-fc113435e140-maas-api-tls\") pod \"maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4\" (UID: \"99717fb1-9461-43dd-81bd-fc113435e140\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" Sep 09 02:45:16.868213 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.868169 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99717fb1-9461-43dd-81bd-fc113435e140-metrics-certs\") pod \"maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4\" (UID: \"99717fb1-9461-43dd-81bd-fc113435e140\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" Sep 09 02:45:16.873894 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.873875 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fwnf\" (UniqueName: \"kubernetes.io/projected/99717fb1-9461-43dd-81bd-fc113435e140-kube-api-access-9fwnf\") pod \"maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4\" (UID: \"99717fb1-9461-43dd-81bd-fc113435e140\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" Sep 09 02:45:16.876636 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:16.876602 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:17.006848 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:17.006731 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" Sep 09 02:45:17.238548 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:17.238501 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:17.282110 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:17.282083 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4"] Sep 09 02:45:17.283857 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:45:17.283801 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99717fb1_9461_43dd_81bd_fc113435e140.slice/crio-c7295cfc04f688e934ad652a406689ac8c69c822e905e2340bbce08432a8b6d6 WatchSource:0}: Error finding container c7295cfc04f688e934ad652a406689ac8c69c822e905e2340bbce08432a8b6d6: Status 404 returned error can't find the container with id c7295cfc04f688e934ad652a406689ac8c69c822e905e2340bbce08432a8b6d6 Sep 09 02:45:17.339877 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:17.339845 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:17.761965 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:17.761932 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:17.789998 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:17.789955 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" event={"ID":"99717fb1-9461-43dd-81bd-fc113435e140","Type":"ContainerStarted","Data":"9ca2f6b01e00b45285d308824d7e38db6d2c9103f8e1d365553a5cca097888f8"} Sep 09 02:45:17.790199 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:17.790053 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" event={"ID":"99717fb1-9461-43dd-81bd-fc113435e140","Type":"ContainerStarted","Data":"c7295cfc04f688e934ad652a406689ac8c69c822e905e2340bbce08432a8b6d6"} Sep 09 02:45:17.790199 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:17.790106 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" Sep 09 02:45:17.811825 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:17.811749 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" podStartSLOduration=1.450386817 podStartE2EDuration="1.81172885s" totalImagesPullingTime="361.342033ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:45:17.809252064 +0000 UTC m=+905.939494165" watchObservedRunningTime="2026-09-09 02:45:17.81172885 +0000 UTC m=+905.941970946" Sep 09 02:45:18.558261 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:18.558218 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:19.777945 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:19.777911 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" Sep 09 02:45:21.865216 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:21.865152 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:22.027300 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:22.027266 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:22.135729 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:22.135694 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:22.232926 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:22.232876 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:22.333531 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:22.333496 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:22.794589 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:22.794559 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" Sep 09 02:45:24.806091 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:24.806057 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" Sep 09 02:45:24.905153 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:24.905116 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:26.544966 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:26.544934 2577 patch_prober.go:28] interesting pod/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": read tcp 10.133.0.2:58786->10.133.0.41:15021: read: connection reset by peer" start-of-body= Sep 09 02:45:26.545398 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:26.545017 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" podUID="aee01b31-2432-4494-a93c-20cd8e73fefa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": read tcp 10.133.0.2:58786->10.133.0.41:15021: read: connection reset by peer" Sep 09 02:45:26.547330 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:26.547289 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:26.842771 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:26.842673 2577 generic.go:356] "Generic (PLEG): container finished" podID="aee01b31-2432-4494-a93c-20cd8e73fefa" containerID="3809d0a7eac02e3f1c90d1ebe8a102e89da24878ebf2656746657e6df150c042" exitCode=0 Sep 09 02:45:26.842771 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:26.842748 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" event={"ID":"aee01b31-2432-4494-a93c-20cd8e73fefa","Type":"ContainerDied","Data":"3809d0a7eac02e3f1c90d1ebe8a102e89da24878ebf2656746657e6df150c042"} Sep 09 02:45:26.843204 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:26.843190 2577 scope.go:117] "RemoveContainer" containerID="3809d0a7eac02e3f1c90d1ebe8a102e89da24878ebf2656746657e6df150c042" Sep 09 02:45:26.844242 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:26.844218 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:45:26.844307 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:26.844285 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:45:26.844346 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:26.844316 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:45:27.850117 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:27.850083 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" event={"ID":"aee01b31-2432-4494-a93c-20cd8e73fefa","Type":"ContainerStarted","Data":"4cb7cc8052fc295ade6473ab98b1575716f29045e34c41435f8a766cf53b7ec1"} Sep 09 02:45:27.931318 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:27.931280 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:28.373470 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:28.373423 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:28.483257 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:28.483222 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:28.575053 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:28.575009 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:28.737159 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:28.737109 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:28.737406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:28.737370 2577 patch_prober.go:28] interesting pod/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Sep 09 02:45:28.737500 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:28.737440 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" podUID="aee01b31-2432-4494-a93c-20cd8e73fefa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Sep 09 02:45:28.756920 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:28.756653 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:29.736936 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:29.736902 2577 patch_prober.go:28] interesting pod/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Sep 09 02:45:29.737342 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:29.736988 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" podUID="aee01b31-2432-4494-a93c-20cd8e73fefa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Sep 09 02:45:30.737253 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:30.737211 2577 patch_prober.go:28] interesting pod/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Sep 09 02:45:30.737730 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:30.737275 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" podUID="aee01b31-2432-4494-a93c-20cd8e73fefa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Sep 09 02:45:31.736810 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:31.736773 2577 patch_prober.go:28] interesting pod/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Sep 09 02:45:31.737002 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:31.736855 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" podUID="aee01b31-2432-4494-a93c-20cd8e73fefa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Sep 09 02:45:32.736784 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:32.736749 2577 patch_prober.go:28] interesting pod/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Sep 09 02:45:32.737223 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:32.736804 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" podUID="aee01b31-2432-4494-a93c-20cd8e73fefa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Sep 09 02:45:32.959380 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:32.959345 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:33.740317 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:33.740283 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:33.740712 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:33.740659 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:33.741244 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:33.741229 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:45:35.201482 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:35.201447 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:35.264724 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:35.264688 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:35.620872 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:35.620770 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:35.943981 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:35.943945 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:36.260863 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:36.260759 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:36.497174 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:36.497146 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:36.703590 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:36.703547 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:38.268375 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:38.268345 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:39.257450 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:39.257420 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:39.344304 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:39.344247 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:39.762424 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:39.762384 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:40.335189 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:40.335159 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:41.323318 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:41.323282 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df"] Sep 09 02:45:41.327642 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:41.327617 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" Sep 09 02:45:41.330306 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:41.330284 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6b11a2-serving-cert\"" Sep 09 02:45:41.339065 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:41.339033 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df"] Sep 09 02:45:41.420837 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:41.420784 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/974e5b86-02e4-42b9-a1ad-ab85414ec680-metrics-certs\") pod \"maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df\" (UID: \"974e5b86-02e4-42b9-a1ad-ab85414ec680\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" Sep 09 02:45:41.421038 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:41.420890 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccr99\" (UniqueName: \"kubernetes.io/projected/974e5b86-02e4-42b9-a1ad-ab85414ec680-kube-api-access-ccr99\") pod \"maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df\" (UID: \"974e5b86-02e4-42b9-a1ad-ab85414ec680\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" Sep 09 02:45:41.421038 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:41.420920 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/974e5b86-02e4-42b9-a1ad-ab85414ec680-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df\" (UID: \"974e5b86-02e4-42b9-a1ad-ab85414ec680\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" Sep 09 02:45:41.421117 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:41.421093 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/974e5b86-02e4-42b9-a1ad-ab85414ec680-maas-api-tls\") pod \"maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df\" (UID: \"974e5b86-02e4-42b9-a1ad-ab85414ec680\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" Sep 09 02:45:41.451432 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:41.451398 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:41.521978 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:41.521931 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccr99\" (UniqueName: \"kubernetes.io/projected/974e5b86-02e4-42b9-a1ad-ab85414ec680-kube-api-access-ccr99\") pod \"maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df\" (UID: \"974e5b86-02e4-42b9-a1ad-ab85414ec680\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" Sep 09 02:45:41.521978 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:41.521982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/974e5b86-02e4-42b9-a1ad-ab85414ec680-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df\" (UID: \"974e5b86-02e4-42b9-a1ad-ab85414ec680\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" Sep 09 02:45:41.522203 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:41.522046 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/974e5b86-02e4-42b9-a1ad-ab85414ec680-maas-api-tls\") pod \"maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df\" (UID: \"974e5b86-02e4-42b9-a1ad-ab85414ec680\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" Sep 09 02:45:41.522203 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:41.522109 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/974e5b86-02e4-42b9-a1ad-ab85414ec680-metrics-certs\") pod \"maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df\" (UID: \"974e5b86-02e4-42b9-a1ad-ab85414ec680\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" Sep 09 02:45:41.522309 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:45:41.522218 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-serving-cert: secret "maas-api-e2e-inf-b-6b11a2-serving-cert" not found Sep 09 02:45:41.522309 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:45:41.522278 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/974e5b86-02e4-42b9-a1ad-ab85414ec680-maas-api-tls podName:974e5b86-02e4-42b9-a1ad-ab85414ec680 nodeName:}" failed. No retries permitted until 2026-09-09 02:45:42.022262825 +0000 UTC m=+930.152504899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/974e5b86-02e4-42b9-a1ad-ab85414ec680-maas-api-tls") pod "maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" (UID: "974e5b86-02e4-42b9-a1ad-ab85414ec680") : secret "maas-api-e2e-inf-b-6b11a2-serving-cert" not found Sep 09 02:45:41.522841 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:41.522801 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/974e5b86-02e4-42b9-a1ad-ab85414ec680-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df\" (UID: \"974e5b86-02e4-42b9-a1ad-ab85414ec680\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" Sep 09 02:45:41.524639 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:41.524616 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/974e5b86-02e4-42b9-a1ad-ab85414ec680-metrics-certs\") pod \"maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df\" (UID: \"974e5b86-02e4-42b9-a1ad-ab85414ec680\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" Sep 09 02:45:41.536981 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:41.536959 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccr99\" (UniqueName: \"kubernetes.io/projected/974e5b86-02e4-42b9-a1ad-ab85414ec680-kube-api-access-ccr99\") pod \"maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df\" (UID: \"974e5b86-02e4-42b9-a1ad-ab85414ec680\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" Sep 09 02:45:41.551874 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:41.551848 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:41.676787 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:41.676747 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:41.878842 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:41.878182 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:41.982513 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:41.982405 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:42.026957 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.026924 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/974e5b86-02e4-42b9-a1ad-ab85414ec680-maas-api-tls\") pod \"maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df\" (UID: \"974e5b86-02e4-42b9-a1ad-ab85414ec680\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" Sep 09 02:45:42.029289 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.029264 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/974e5b86-02e4-42b9-a1ad-ab85414ec680-maas-api-tls\") pod \"maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df\" (UID: \"974e5b86-02e4-42b9-a1ad-ab85414ec680\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" Sep 09 02:45:42.078092 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.078047 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk"] Sep 09 02:45:42.082685 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.082661 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk" Sep 09 02:45:42.085395 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.085370 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6b11a2\"" Sep 09 02:45:42.085853 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.085812 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6b11a2-dockercfg-hhk6s\"" Sep 09 02:45:42.087777 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.087756 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk"] Sep 09 02:45:42.127787 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.127756 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9zzx\" (UniqueName: \"kubernetes.io/projected/717f071e-8c75-454f-96f4-8e42ca69738b-kube-api-access-h9zzx\") pod \"payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk\" (UID: \"717f071e-8c75-454f-96f4-8e42ca69738b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk" Sep 09 02:45:42.127971 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.127865 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/717f071e-8c75-454f-96f4-8e42ca69738b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk\" (UID: \"717f071e-8c75-454f-96f4-8e42ca69738b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk" Sep 09 02:45:42.228740 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.228698 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9zzx\" (UniqueName: \"kubernetes.io/projected/717f071e-8c75-454f-96f4-8e42ca69738b-kube-api-access-h9zzx\") pod \"payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk\" (UID: \"717f071e-8c75-454f-96f4-8e42ca69738b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk" Sep 09 02:45:42.228740 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.228743 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/717f071e-8c75-454f-96f4-8e42ca69738b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk\" (UID: \"717f071e-8c75-454f-96f4-8e42ca69738b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk" Sep 09 02:45:42.229527 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.229507 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/717f071e-8c75-454f-96f4-8e42ca69738b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk\" (UID: \"717f071e-8c75-454f-96f4-8e42ca69738b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk" Sep 09 02:45:42.237078 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.237015 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9zzx\" (UniqueName: \"kubernetes.io/projected/717f071e-8c75-454f-96f4-8e42ca69738b-kube-api-access-h9zzx\") pod \"payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk\" (UID: \"717f071e-8c75-454f-96f4-8e42ca69738b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk" Sep 09 02:45:42.241918 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.241895 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" Sep 09 02:45:42.334313 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.334278 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:42.395573 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.395542 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk" Sep 09 02:45:42.539202 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.539184 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df"] Sep 09 02:45:42.541639 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:45:42.541610 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod974e5b86_02e4_42b9_a1ad_ab85414ec680.slice/crio-a55545352f8bbc0a45871af99216f9c352fe96d087bb1b17b8ab491345e62c73 WatchSource:0}: Error finding container a55545352f8bbc0a45871af99216f9c352fe96d087bb1b17b8ab491345e62c73: Status 404 returned error can't find the container with id a55545352f8bbc0a45871af99216f9c352fe96d087bb1b17b8ab491345e62c73 Sep 09 02:45:42.673924 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.673869 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk"] Sep 09 02:45:42.683204 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:45:42.683177 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717f071e_8c75_454f_96f4_8e42ca69738b.slice/crio-375d869e3aedf54a0de466776438cc59798b7b615d36f93ced8d509a40ac3960 WatchSource:0}: Error finding container 375d869e3aedf54a0de466776438cc59798b7b615d36f93ced8d509a40ac3960: Status 404 returned error can't find the container with id 375d869e3aedf54a0de466776438cc59798b7b615d36f93ced8d509a40ac3960 Sep 09 02:45:42.940747 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.940723 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" event={"ID":"974e5b86-02e4-42b9-a1ad-ab85414ec680","Type":"ContainerStarted","Data":"a55545352f8bbc0a45871af99216f9c352fe96d087bb1b17b8ab491345e62c73"} Sep 09 02:45:42.942495 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.942467 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk" event={"ID":"717f071e-8c75-454f-96f4-8e42ca69738b","Type":"ContainerStarted","Data":"1104c6ef0d10a0689656755173c04647ae5726a91fc9c295b676add948f99733"} Sep 09 02:45:42.942628 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.942531 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk" event={"ID":"717f071e-8c75-454f-96f4-8e42ca69738b","Type":"ContainerStarted","Data":"375d869e3aedf54a0de466776438cc59798b7b615d36f93ced8d509a40ac3960"} Sep 09 02:45:42.942628 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.942568 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk" Sep 09 02:45:42.963982 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:42.963914 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk" podStartSLOduration=0.963895481 podStartE2EDuration="963.895481ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:45:42.959854485 +0000 UTC m=+931.090096582" watchObservedRunningTime="2026-09-09 02:45:42.963895481 +0000 UTC m=+931.094137579" Sep 09 02:45:43.252923 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:43.252795 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:43.949982 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:43.949943 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" event={"ID":"974e5b86-02e4-42b9-a1ad-ab85414ec680","Type":"ContainerStarted","Data":"ab44afa535d73980dd38b6f2308a5d440aa5e7ddfb2b4154a035fe1a79712774"} Sep 09 02:45:43.969424 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:43.969360 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" podStartSLOduration=2.631365694 podStartE2EDuration="2.969337243s" totalImagesPullingTime="337.971549ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:45:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:45:43.967952317 +0000 UTC m=+932.098194425" watchObservedRunningTime="2026-09-09 02:45:43.969337243 +0000 UTC m=+932.099579344" Sep 09 02:45:44.955517 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:44.955484 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" Sep 09 02:45:48.979861 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:48.979805 2577 generic.go:356] "Generic (PLEG): container finished" podID="0b5deaea-bded-4181-99aa-69d4a867daaa" containerID="2e7d71b73df0a64a36efd486f29a2605e3f63c8ecb4ae9743e7b0b113db4bb3a" exitCode=0 Sep 09 02:45:48.980407 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:48.979880 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" event={"ID":"0b5deaea-bded-4181-99aa-69d4a867daaa","Type":"ContainerDied","Data":"2e7d71b73df0a64a36efd486f29a2605e3f63c8ecb4ae9743e7b0b113db4bb3a"} Sep 09 02:45:48.980407 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:48.980311 2577 scope.go:117] "RemoveContainer" containerID="2e7d71b73df0a64a36efd486f29a2605e3f63c8ecb4ae9743e7b0b113db4bb3a" Sep 09 02:45:48.981398 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:48.981372 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:45:48.981451 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:48.981440 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:45:48.981487 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:48.981472 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:45:49.988162 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:49.988132 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" event={"ID":"0b5deaea-bded-4181-99aa-69d4a867daaa","Type":"ContainerStarted","Data":"3b8dbc91e35076b90e6b9fb1baac506f4cea8fcbb779b921fc9d1c2726221b67"} Sep 09 02:45:50.087597 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:50.087557 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:50.087918 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:50.087889 2577 patch_prober.go:28] interesting pod/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 09 02:45:50.087983 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:50.087948 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" podUID="0b5deaea-bded-4181-99aa-69d4a867daaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 09 02:45:50.975105 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:50.975060 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" Sep 09 02:45:51.087322 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:51.087281 2577 patch_prober.go:28] interesting pod/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 09 02:45:51.087714 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:51.087372 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" podUID="0b5deaea-bded-4181-99aa-69d4a867daaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 09 02:45:52.086907 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:52.086863 2577 patch_prober.go:28] interesting pod/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 09 02:45:52.087093 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:52.086931 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" podUID="0b5deaea-bded-4181-99aa-69d4a867daaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 09 02:45:53.086831 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:53.086790 2577 patch_prober.go:28] interesting pod/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 09 02:45:53.087228 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:53.086878 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" podUID="0b5deaea-bded-4181-99aa-69d4a867daaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 09 02:45:53.953291 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:53.953248 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk" Sep 09 02:45:54.087735 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:54.087697 2577 patch_prober.go:28] interesting pod/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 09 02:45:54.088169 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:54.087753 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" podUID="0b5deaea-bded-4181-99aa-69d4a867daaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 09 02:45:55.091241 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:55.091209 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:55.091657 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:55.091499 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:55.092158 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:55.092141 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:45:55.846369 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:55.846333 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:56.414069 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:56.414032 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95"] Sep 09 02:45:56.414577 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:56.414287 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" podUID="c5653c22-2a40-465a-888f-2aae14e21d39" containerName="maas-api" containerID="cri-o://823e3bcf67c42c5003ad2ba8c13a7ad86833e29d3a3322df5649cb4115199091" gracePeriod=30 Sep 09 02:45:56.667559 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:56.667499 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" Sep 09 02:45:56.798650 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:56.798610 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdv9b\" (UniqueName: \"kubernetes.io/projected/c5653c22-2a40-465a-888f-2aae14e21d39-kube-api-access-sdv9b\") pod \"c5653c22-2a40-465a-888f-2aae14e21d39\" (UID: \"c5653c22-2a40-465a-888f-2aae14e21d39\") " Sep 09 02:45:56.798836 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:56.798667 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5653c22-2a40-465a-888f-2aae14e21d39-openshift-service-ca\") pod \"c5653c22-2a40-465a-888f-2aae14e21d39\" (UID: \"c5653c22-2a40-465a-888f-2aae14e21d39\") " Sep 09 02:45:56.798836 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:56.798708 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5653c22-2a40-465a-888f-2aae14e21d39-maas-api-tls\") pod \"c5653c22-2a40-465a-888f-2aae14e21d39\" (UID: \"c5653c22-2a40-465a-888f-2aae14e21d39\") " Sep 09 02:45:56.798836 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:56.798791 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5653c22-2a40-465a-888f-2aae14e21d39-metrics-certs\") pod \"c5653c22-2a40-465a-888f-2aae14e21d39\" (UID: \"c5653c22-2a40-465a-888f-2aae14e21d39\") " Sep 09 02:45:56.799112 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:56.799082 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5653c22-2a40-465a-888f-2aae14e21d39-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c5653c22-2a40-465a-888f-2aae14e21d39" (UID: "c5653c22-2a40-465a-888f-2aae14e21d39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:45:56.801188 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:56.801157 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5653c22-2a40-465a-888f-2aae14e21d39-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c5653c22-2a40-465a-888f-2aae14e21d39" (UID: "c5653c22-2a40-465a-888f-2aae14e21d39"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:45:56.801188 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:56.801170 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5653c22-2a40-465a-888f-2aae14e21d39-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c5653c22-2a40-465a-888f-2aae14e21d39" (UID: "c5653c22-2a40-465a-888f-2aae14e21d39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:45:56.801366 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:56.801234 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5653c22-2a40-465a-888f-2aae14e21d39-kube-api-access-sdv9b" (OuterVolumeSpecName: "kube-api-access-sdv9b") pod "c5653c22-2a40-465a-888f-2aae14e21d39" (UID: "c5653c22-2a40-465a-888f-2aae14e21d39"). InnerVolumeSpecName "kube-api-access-sdv9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:45:56.900128 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:56.900092 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdv9b\" (UniqueName: \"kubernetes.io/projected/c5653c22-2a40-465a-888f-2aae14e21d39-kube-api-access-sdv9b\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:45:56.900128 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:56.900126 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5653c22-2a40-465a-888f-2aae14e21d39-openshift-service-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:45:56.900338 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:56.900137 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5653c22-2a40-465a-888f-2aae14e21d39-maas-api-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:45:56.900338 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:56.900146 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5653c22-2a40-465a-888f-2aae14e21d39-metrics-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:45:57.029456 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:57.029367 2577 generic.go:356] "Generic (PLEG): container finished" podID="c5653c22-2a40-465a-888f-2aae14e21d39" containerID="823e3bcf67c42c5003ad2ba8c13a7ad86833e29d3a3322df5649cb4115199091" exitCode=0 Sep 09 02:45:57.029456 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:57.029417 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" Sep 09 02:45:57.029647 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:57.029455 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" event={"ID":"c5653c22-2a40-465a-888f-2aae14e21d39","Type":"ContainerDied","Data":"823e3bcf67c42c5003ad2ba8c13a7ad86833e29d3a3322df5649cb4115199091"} Sep 09 02:45:57.029647 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:57.029539 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95" event={"ID":"c5653c22-2a40-465a-888f-2aae14e21d39","Type":"ContainerDied","Data":"51ee4ad33e7a76aad1ed42153f74acffb92bfdf0b7b71a43454671d81e024b02"} Sep 09 02:45:57.029647 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:57.029565 2577 scope.go:117] "RemoveContainer" containerID="823e3bcf67c42c5003ad2ba8c13a7ad86833e29d3a3322df5649cb4115199091" Sep 09 02:45:57.039739 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:57.039718 2577 scope.go:117] "RemoveContainer" containerID="823e3bcf67c42c5003ad2ba8c13a7ad86833e29d3a3322df5649cb4115199091" Sep 09 02:45:57.040056 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:45:57.040034 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"823e3bcf67c42c5003ad2ba8c13a7ad86833e29d3a3322df5649cb4115199091\": container with ID starting with 823e3bcf67c42c5003ad2ba8c13a7ad86833e29d3a3322df5649cb4115199091 not found: ID does not exist" containerID="823e3bcf67c42c5003ad2ba8c13a7ad86833e29d3a3322df5649cb4115199091" Sep 09 02:45:57.040115 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:57.040064 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823e3bcf67c42c5003ad2ba8c13a7ad86833e29d3a3322df5649cb4115199091"} err="failed to get container status \"823e3bcf67c42c5003ad2ba8c13a7ad86833e29d3a3322df5649cb4115199091\": rpc error: code = NotFound desc = could not find container \"823e3bcf67c42c5003ad2ba8c13a7ad86833e29d3a3322df5649cb4115199091\": container with ID starting with 823e3bcf67c42c5003ad2ba8c13a7ad86833e29d3a3322df5649cb4115199091 not found: ID does not exist" Sep 09 02:45:57.046268 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:57.046247 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:57.053622 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:57.053598 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95"] Sep 09 02:45:57.058264 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:57.058240 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-79a2702e-74cc7764fb-6bq95"] Sep 09 02:45:57.241373 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:57.241339 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:57.343200 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:57.343098 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:57.441348 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:57.441315 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:57.745382 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:57.745350 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:57.949368 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:57.949335 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:45:58.309943 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:45:58.309908 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5653c22-2a40-465a-888f-2aae14e21d39" path="/var/lib/kubelet/pods/c5653c22-2a40-465a-888f-2aae14e21d39/volumes" Sep 09 02:46:00.766838 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:00.766774 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:00.863400 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:00.863357 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:00.879430 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:00.879396 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb"] Sep 09 02:46:00.880086 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:00.880061 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5653c22-2a40-465a-888f-2aae14e21d39" containerName="maas-api" Sep 09 02:46:00.880086 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:00.880083 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5653c22-2a40-465a-888f-2aae14e21d39" containerName="maas-api" Sep 09 02:46:00.880294 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:00.880180 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5653c22-2a40-465a-888f-2aae14e21d39" containerName="maas-api" Sep 09 02:46:00.883359 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:00.883334 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" Sep 09 02:46:00.886211 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:00.886183 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b0df6c69-serving-cert\"" Sep 09 02:46:00.893714 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:00.893512 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb"] Sep 09 02:46:00.963144 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:00.963110 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:01.042342 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:01.042245 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c385eed-1597-4ed1-8c58-47eb0f3b164d-maas-api-tls\") pod \"maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb\" (UID: \"9c385eed-1597-4ed1-8c58-47eb0f3b164d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" Sep 09 02:46:01.042342 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:01.042280 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbnvq\" (UniqueName: \"kubernetes.io/projected/9c385eed-1597-4ed1-8c58-47eb0f3b164d-kube-api-access-sbnvq\") pod \"maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb\" (UID: \"9c385eed-1597-4ed1-8c58-47eb0f3b164d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" Sep 09 02:46:01.042342 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:01.042307 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c385eed-1597-4ed1-8c58-47eb0f3b164d-openshift-service-ca\") pod \"maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb\" (UID: \"9c385eed-1597-4ed1-8c58-47eb0f3b164d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" Sep 09 02:46:01.042586 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:01.042393 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c385eed-1597-4ed1-8c58-47eb0f3b164d-metrics-certs\") pod \"maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb\" (UID: \"9c385eed-1597-4ed1-8c58-47eb0f3b164d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" Sep 09 02:46:01.073353 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:01.073317 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:01.143384 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:01.143347 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c385eed-1597-4ed1-8c58-47eb0f3b164d-maas-api-tls\") pod \"maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb\" (UID: \"9c385eed-1597-4ed1-8c58-47eb0f3b164d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" Sep 09 02:46:01.143384 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:01.143385 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbnvq\" (UniqueName: \"kubernetes.io/projected/9c385eed-1597-4ed1-8c58-47eb0f3b164d-kube-api-access-sbnvq\") pod \"maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb\" (UID: \"9c385eed-1597-4ed1-8c58-47eb0f3b164d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" Sep 09 02:46:01.143656 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:01.143408 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c385eed-1597-4ed1-8c58-47eb0f3b164d-openshift-service-ca\") pod \"maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb\" (UID: \"9c385eed-1597-4ed1-8c58-47eb0f3b164d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" Sep 09 02:46:01.143656 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:01.143433 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c385eed-1597-4ed1-8c58-47eb0f3b164d-metrics-certs\") pod \"maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb\" (UID: \"9c385eed-1597-4ed1-8c58-47eb0f3b164d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" Sep 09 02:46:01.144125 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:01.144097 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c385eed-1597-4ed1-8c58-47eb0f3b164d-openshift-service-ca\") pod \"maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb\" (UID: \"9c385eed-1597-4ed1-8c58-47eb0f3b164d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" Sep 09 02:46:01.145788 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:01.145766 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c385eed-1597-4ed1-8c58-47eb0f3b164d-maas-api-tls\") pod \"maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb\" (UID: \"9c385eed-1597-4ed1-8c58-47eb0f3b164d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" Sep 09 02:46:01.145877 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:01.145850 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c385eed-1597-4ed1-8c58-47eb0f3b164d-metrics-certs\") pod \"maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb\" (UID: \"9c385eed-1597-4ed1-8c58-47eb0f3b164d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" Sep 09 02:46:01.154139 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:01.154113 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbnvq\" (UniqueName: \"kubernetes.io/projected/9c385eed-1597-4ed1-8c58-47eb0f3b164d-kube-api-access-sbnvq\") pod \"maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb\" (UID: \"9c385eed-1597-4ed1-8c58-47eb0f3b164d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" Sep 09 02:46:01.169055 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:01.169024 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:01.197657 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:01.197624 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" Sep 09 02:46:01.438353 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:01.438307 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:01.478088 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:01.478061 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb"] Sep 09 02:46:01.479987 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:46:01.479954 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c385eed_1597_4ed1_8c58_47eb0f3b164d.slice/crio-dfbd404fc3e437cb7290f88b1a1f45b3c7474d754a4f09503bf897c2ba7bda7b WatchSource:0}: Error finding container dfbd404fc3e437cb7290f88b1a1f45b3c7474d754a4f09503bf897c2ba7bda7b: Status 404 returned error can't find the container with id dfbd404fc3e437cb7290f88b1a1f45b3c7474d754a4f09503bf897c2ba7bda7b Sep 09 02:46:01.546678 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:01.546625 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:02.057873 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:02.057830 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" event={"ID":"9c385eed-1597-4ed1-8c58-47eb0f3b164d","Type":"ContainerStarted","Data":"531be42df86e5bc0ba5c54b5dcfa4f458f06a643add602413c689ae31ae4d163"} Sep 09 02:46:02.058322 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:02.057901 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" event={"ID":"9c385eed-1597-4ed1-8c58-47eb0f3b164d","Type":"ContainerStarted","Data":"dfbd404fc3e437cb7290f88b1a1f45b3c7474d754a4f09503bf897c2ba7bda7b"} Sep 09 02:46:02.058322 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:02.058025 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" Sep 09 02:46:02.078827 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:02.078756 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" podStartSLOduration=1.6886383029999998 podStartE2EDuration="2.078737218s" totalImagesPullingTime="390.098915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:46:02.076557199 +0000 UTC m=+950.206799295" watchObservedRunningTime="2026-09-09 02:46:02.078737218 +0000 UTC m=+950.208979314" Sep 09 02:46:02.496781 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:02.496737 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:04.114706 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:04.114671 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:08.069196 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:08.069164 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" Sep 09 02:46:21.957465 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:21.957428 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:27.674501 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:27.674469 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:31.625534 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:31.625500 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb"] Sep 09 02:46:31.626044 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:31.625771 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" podUID="9c385eed-1597-4ed1-8c58-47eb0f3b164d" containerName="maas-api" containerID="cri-o://531be42df86e5bc0ba5c54b5dcfa4f458f06a643add602413c689ae31ae4d163" gracePeriod=30 Sep 09 02:46:31.658750 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:31.658723 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:31.745180 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:31.745144 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:31.811531 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:31.811498 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:31.880786 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:31.880762 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" Sep 09 02:46:31.892199 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:31.892118 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:31.964498 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:31.964464 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:32.039875 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.039806 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:32.060959 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.060927 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c385eed-1597-4ed1-8c58-47eb0f3b164d-openshift-service-ca\") pod \"9c385eed-1597-4ed1-8c58-47eb0f3b164d\" (UID: \"9c385eed-1597-4ed1-8c58-47eb0f3b164d\") " Sep 09 02:46:32.061127 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.060975 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbnvq\" (UniqueName: \"kubernetes.io/projected/9c385eed-1597-4ed1-8c58-47eb0f3b164d-kube-api-access-sbnvq\") pod \"9c385eed-1597-4ed1-8c58-47eb0f3b164d\" (UID: \"9c385eed-1597-4ed1-8c58-47eb0f3b164d\") " Sep 09 02:46:32.061127 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.061080 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c385eed-1597-4ed1-8c58-47eb0f3b164d-metrics-certs\") pod \"9c385eed-1597-4ed1-8c58-47eb0f3b164d\" (UID: \"9c385eed-1597-4ed1-8c58-47eb0f3b164d\") " Sep 09 02:46:32.061127 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.061115 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c385eed-1597-4ed1-8c58-47eb0f3b164d-maas-api-tls\") pod \"9c385eed-1597-4ed1-8c58-47eb0f3b164d\" (UID: \"9c385eed-1597-4ed1-8c58-47eb0f3b164d\") " Sep 09 02:46:32.061370 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.061338 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c385eed-1597-4ed1-8c58-47eb0f3b164d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9c385eed-1597-4ed1-8c58-47eb0f3b164d" (UID: "9c385eed-1597-4ed1-8c58-47eb0f3b164d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:46:32.063228 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.063198 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c385eed-1597-4ed1-8c58-47eb0f3b164d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9c385eed-1597-4ed1-8c58-47eb0f3b164d" (UID: "9c385eed-1597-4ed1-8c58-47eb0f3b164d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:46:32.063339 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.063223 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c385eed-1597-4ed1-8c58-47eb0f3b164d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9c385eed-1597-4ed1-8c58-47eb0f3b164d" (UID: "9c385eed-1597-4ed1-8c58-47eb0f3b164d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:46:32.063339 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.063263 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c385eed-1597-4ed1-8c58-47eb0f3b164d-kube-api-access-sbnvq" (OuterVolumeSpecName: "kube-api-access-sbnvq") pod "9c385eed-1597-4ed1-8c58-47eb0f3b164d" (UID: "9c385eed-1597-4ed1-8c58-47eb0f3b164d"). InnerVolumeSpecName "kube-api-access-sbnvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:46:32.163022 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.162917 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c385eed-1597-4ed1-8c58-47eb0f3b164d-openshift-service-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:46:32.163022 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.162958 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbnvq\" (UniqueName: \"kubernetes.io/projected/9c385eed-1597-4ed1-8c58-47eb0f3b164d-kube-api-access-sbnvq\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:46:32.163022 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.162971 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c385eed-1597-4ed1-8c58-47eb0f3b164d-metrics-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:46:32.163022 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.162982 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c385eed-1597-4ed1-8c58-47eb0f3b164d-maas-api-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:46:32.238903 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.238866 2577 generic.go:356] "Generic (PLEG): container finished" podID="9c385eed-1597-4ed1-8c58-47eb0f3b164d" containerID="531be42df86e5bc0ba5c54b5dcfa4f458f06a643add602413c689ae31ae4d163" exitCode=0 Sep 09 02:46:32.239088 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.238914 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" Sep 09 02:46:32.239088 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.238950 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" event={"ID":"9c385eed-1597-4ed1-8c58-47eb0f3b164d","Type":"ContainerDied","Data":"531be42df86e5bc0ba5c54b5dcfa4f458f06a643add602413c689ae31ae4d163"} Sep 09 02:46:32.239088 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.239001 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb" event={"ID":"9c385eed-1597-4ed1-8c58-47eb0f3b164d","Type":"ContainerDied","Data":"dfbd404fc3e437cb7290f88b1a1f45b3c7474d754a4f09503bf897c2ba7bda7b"} Sep 09 02:46:32.239088 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.239024 2577 scope.go:117] "RemoveContainer" containerID="531be42df86e5bc0ba5c54b5dcfa4f458f06a643add602413c689ae31ae4d163" Sep 09 02:46:32.249219 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.249202 2577 scope.go:117] "RemoveContainer" containerID="531be42df86e5bc0ba5c54b5dcfa4f458f06a643add602413c689ae31ae4d163" Sep 09 02:46:32.249510 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:46:32.249489 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"531be42df86e5bc0ba5c54b5dcfa4f458f06a643add602413c689ae31ae4d163\": container with ID starting with 531be42df86e5bc0ba5c54b5dcfa4f458f06a643add602413c689ae31ae4d163 not found: ID does not exist" containerID="531be42df86e5bc0ba5c54b5dcfa4f458f06a643add602413c689ae31ae4d163" Sep 09 02:46:32.249573 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.249523 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531be42df86e5bc0ba5c54b5dcfa4f458f06a643add602413c689ae31ae4d163"} err="failed to get container status \"531be42df86e5bc0ba5c54b5dcfa4f458f06a643add602413c689ae31ae4d163\": rpc error: code = NotFound desc = could not find container \"531be42df86e5bc0ba5c54b5dcfa4f458f06a643add602413c689ae31ae4d163\": container with ID starting with 531be42df86e5bc0ba5c54b5dcfa4f458f06a643add602413c689ae31ae4d163 not found: ID does not exist" Sep 09 02:46:32.263133 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.263105 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb"] Sep 09 02:46:32.266631 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.266607 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b0df6c69-78f449dddb-7wqgb"] Sep 09 02:46:32.311491 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.311461 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c385eed-1597-4ed1-8c58-47eb0f3b164d" path="/var/lib/kubelet/pods/9c385eed-1597-4ed1-8c58-47eb0f3b164d/volumes" Sep 09 02:46:32.459062 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.458981 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:32.772121 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:32.771996 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:33.055679 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:33.055595 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:34.976060 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:34.975961 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:35.073646 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:35.073614 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:36.108908 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.108866 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv"] Sep 09 02:46:36.109554 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.109532 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c385eed-1597-4ed1-8c58-47eb0f3b164d" containerName="maas-api" Sep 09 02:46:36.109639 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.109556 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c385eed-1597-4ed1-8c58-47eb0f3b164d" containerName="maas-api" Sep 09 02:46:36.109698 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.109679 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c385eed-1597-4ed1-8c58-47eb0f3b164d" containerName="maas-api" Sep 09 02:46:36.114282 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.114256 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" Sep 09 02:46:36.116774 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.116745 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7e17db81-serving-cert\"" Sep 09 02:46:36.125265 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.125223 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv"] Sep 09 02:46:36.151023 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.150990 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:36.252976 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.252935 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:36.301407 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.301366 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e14f6b7f-29ac-43a4-8581-ca0841f70721-metrics-certs\") pod \"maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv\" (UID: \"e14f6b7f-29ac-43a4-8581-ca0841f70721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" Sep 09 02:46:36.301627 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.301418 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e14f6b7f-29ac-43a4-8581-ca0841f70721-maas-api-tls\") pod \"maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv\" (UID: \"e14f6b7f-29ac-43a4-8581-ca0841f70721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" Sep 09 02:46:36.301627 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.301522 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e14f6b7f-29ac-43a4-8581-ca0841f70721-openshift-service-ca\") pod \"maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv\" (UID: \"e14f6b7f-29ac-43a4-8581-ca0841f70721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" Sep 09 02:46:36.301627 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.301596 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58mgw\" (UniqueName: \"kubernetes.io/projected/e14f6b7f-29ac-43a4-8581-ca0841f70721-kube-api-access-58mgw\") pod \"maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv\" (UID: \"e14f6b7f-29ac-43a4-8581-ca0841f70721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" Sep 09 02:46:36.403043 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.403003 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e14f6b7f-29ac-43a4-8581-ca0841f70721-metrics-certs\") pod \"maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv\" (UID: \"e14f6b7f-29ac-43a4-8581-ca0841f70721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" Sep 09 02:46:36.403216 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.403049 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e14f6b7f-29ac-43a4-8581-ca0841f70721-maas-api-tls\") pod \"maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv\" (UID: \"e14f6b7f-29ac-43a4-8581-ca0841f70721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" Sep 09 02:46:36.403216 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:46:36.403156 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-serving-cert: secret "maas-api-e2e-ait-7e17db81-serving-cert" not found Sep 09 02:46:36.403296 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:46:36.403224 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e14f6b7f-29ac-43a4-8581-ca0841f70721-maas-api-tls podName:e14f6b7f-29ac-43a4-8581-ca0841f70721 nodeName:}" failed. No retries permitted until 2026-09-09 02:46:36.903206684 +0000 UTC m=+985.033448758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e14f6b7f-29ac-43a4-8581-ca0841f70721-maas-api-tls") pod "maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" (UID: "e14f6b7f-29ac-43a4-8581-ca0841f70721") : secret "maas-api-e2e-ait-7e17db81-serving-cert" not found Sep 09 02:46:36.403296 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.403238 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e14f6b7f-29ac-43a4-8581-ca0841f70721-openshift-service-ca\") pod \"maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv\" (UID: \"e14f6b7f-29ac-43a4-8581-ca0841f70721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" Sep 09 02:46:36.403296 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.403289 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58mgw\" (UniqueName: \"kubernetes.io/projected/e14f6b7f-29ac-43a4-8581-ca0841f70721-kube-api-access-58mgw\") pod \"maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv\" (UID: \"e14f6b7f-29ac-43a4-8581-ca0841f70721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" Sep 09 02:46:36.403981 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.403957 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e14f6b7f-29ac-43a4-8581-ca0841f70721-openshift-service-ca\") pod \"maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv\" (UID: \"e14f6b7f-29ac-43a4-8581-ca0841f70721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" Sep 09 02:46:36.405586 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.405562 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e14f6b7f-29ac-43a4-8581-ca0841f70721-metrics-certs\") pod \"maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv\" (UID: \"e14f6b7f-29ac-43a4-8581-ca0841f70721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" Sep 09 02:46:36.411874 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.411846 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58mgw\" (UniqueName: \"kubernetes.io/projected/e14f6b7f-29ac-43a4-8581-ca0841f70721-kube-api-access-58mgw\") pod \"maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv\" (UID: \"e14f6b7f-29ac-43a4-8581-ca0841f70721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" Sep 09 02:46:36.538098 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.538062 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:36.746216 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.746135 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:36.844061 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.844026 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:36.907752 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.907718 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e14f6b7f-29ac-43a4-8581-ca0841f70721-maas-api-tls\") pod \"maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv\" (UID: \"e14f6b7f-29ac-43a4-8581-ca0841f70721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" Sep 09 02:46:36.910258 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:36.910235 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e14f6b7f-29ac-43a4-8581-ca0841f70721-maas-api-tls\") pod \"maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv\" (UID: \"e14f6b7f-29ac-43a4-8581-ca0841f70721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" Sep 09 02:46:37.027882 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:37.027780 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" Sep 09 02:46:37.305038 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:37.304967 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv"] Sep 09 02:46:37.308078 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:46:37.308048 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14f6b7f_29ac_43a4_8581_ca0841f70721.slice/crio-10215e6139a84f1d40e248c5982263b5127407d71a613e068217599b02d8f0b0 WatchSource:0}: Error finding container 10215e6139a84f1d40e248c5982263b5127407d71a613e068217599b02d8f0b0: Status 404 returned error can't find the container with id 10215e6139a84f1d40e248c5982263b5127407d71a613e068217599b02d8f0b0 Sep 09 02:46:38.274851 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:38.274807 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" event={"ID":"e14f6b7f-29ac-43a4-8581-ca0841f70721","Type":"ContainerStarted","Data":"7442ee653ddd84f7a574b6052f310bff17a9870febf41692667fa11b542d4b92"} Sep 09 02:46:38.275031 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:38.274866 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" event={"ID":"e14f6b7f-29ac-43a4-8581-ca0841f70721","Type":"ContainerStarted","Data":"10215e6139a84f1d40e248c5982263b5127407d71a613e068217599b02d8f0b0"} Sep 09 02:46:38.275031 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:38.274967 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" Sep 09 02:46:38.294344 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:38.294296 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" podStartSLOduration=1.762185865 podStartE2EDuration="2.294281099s" totalImagesPullingTime="532.095234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:46:38.291877849 +0000 UTC m=+986.422119945" watchObservedRunningTime="2026-09-09 02:46:38.294281099 +0000 UTC m=+986.424523194" Sep 09 02:46:38.540347 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:38.540267 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:46:44.286733 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:46:44.286704 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" Sep 09 02:47:02.841368 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:02.841325 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:05.750863 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:05.750828 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:06.313232 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:06.313184 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:06.900268 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:06.900231 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv"] Sep 09 02:47:06.900738 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:06.900540 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" podUID="e14f6b7f-29ac-43a4-8581-ca0841f70721" containerName="maas-api" containerID="cri-o://7442ee653ddd84f7a574b6052f310bff17a9870febf41692667fa11b542d4b92" gracePeriod=30 Sep 09 02:47:06.929106 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:06.929074 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:07.010037 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.009986 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:07.074604 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.074579 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:07.148548 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.148509 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:07.153868 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.153792 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" Sep 09 02:47:07.204244 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.204206 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:07.330302 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.330271 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e14f6b7f-29ac-43a4-8581-ca0841f70721-metrics-certs\") pod \"e14f6b7f-29ac-43a4-8581-ca0841f70721\" (UID: \"e14f6b7f-29ac-43a4-8581-ca0841f70721\") " Sep 09 02:47:07.330487 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.330345 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e14f6b7f-29ac-43a4-8581-ca0841f70721-openshift-service-ca\") pod \"e14f6b7f-29ac-43a4-8581-ca0841f70721\" (UID: \"e14f6b7f-29ac-43a4-8581-ca0841f70721\") " Sep 09 02:47:07.330487 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.330374 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e14f6b7f-29ac-43a4-8581-ca0841f70721-maas-api-tls\") pod \"e14f6b7f-29ac-43a4-8581-ca0841f70721\" (UID: \"e14f6b7f-29ac-43a4-8581-ca0841f70721\") " Sep 09 02:47:07.330487 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.330394 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58mgw\" (UniqueName: \"kubernetes.io/projected/e14f6b7f-29ac-43a4-8581-ca0841f70721-kube-api-access-58mgw\") pod \"e14f6b7f-29ac-43a4-8581-ca0841f70721\" (UID: \"e14f6b7f-29ac-43a4-8581-ca0841f70721\") " Sep 09 02:47:07.330804 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.330764 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e14f6b7f-29ac-43a4-8581-ca0841f70721-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e14f6b7f-29ac-43a4-8581-ca0841f70721" (UID: "e14f6b7f-29ac-43a4-8581-ca0841f70721"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:47:07.332608 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.332562 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14f6b7f-29ac-43a4-8581-ca0841f70721-kube-api-access-58mgw" (OuterVolumeSpecName: "kube-api-access-58mgw") pod "e14f6b7f-29ac-43a4-8581-ca0841f70721" (UID: "e14f6b7f-29ac-43a4-8581-ca0841f70721"). InnerVolumeSpecName "kube-api-access-58mgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:47:07.332608 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.332596 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e14f6b7f-29ac-43a4-8581-ca0841f70721-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e14f6b7f-29ac-43a4-8581-ca0841f70721" (UID: "e14f6b7f-29ac-43a4-8581-ca0841f70721"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:47:07.332608 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.332573 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e14f6b7f-29ac-43a4-8581-ca0841f70721-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e14f6b7f-29ac-43a4-8581-ca0841f70721" (UID: "e14f6b7f-29ac-43a4-8581-ca0841f70721"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:47:07.431338 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.431241 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e14f6b7f-29ac-43a4-8581-ca0841f70721-maas-api-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:07.431338 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.431271 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58mgw\" (UniqueName: \"kubernetes.io/projected/e14f6b7f-29ac-43a4-8581-ca0841f70721-kube-api-access-58mgw\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:07.431338 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.431283 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e14f6b7f-29ac-43a4-8581-ca0841f70721-metrics-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:07.431338 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.431295 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e14f6b7f-29ac-43a4-8581-ca0841f70721-openshift-service-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:07.436361 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.436330 2577 generic.go:356] "Generic (PLEG): container finished" podID="e14f6b7f-29ac-43a4-8581-ca0841f70721" containerID="7442ee653ddd84f7a574b6052f310bff17a9870febf41692667fa11b542d4b92" exitCode=0 Sep 09 02:47:07.436488 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.436375 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" Sep 09 02:47:07.436488 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.436387 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" event={"ID":"e14f6b7f-29ac-43a4-8581-ca0841f70721","Type":"ContainerDied","Data":"7442ee653ddd84f7a574b6052f310bff17a9870febf41692667fa11b542d4b92"} Sep 09 02:47:07.436488 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.436428 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv" event={"ID":"e14f6b7f-29ac-43a4-8581-ca0841f70721","Type":"ContainerDied","Data":"10215e6139a84f1d40e248c5982263b5127407d71a613e068217599b02d8f0b0"} Sep 09 02:47:07.436488 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.436449 2577 scope.go:117] "RemoveContainer" containerID="7442ee653ddd84f7a574b6052f310bff17a9870febf41692667fa11b542d4b92" Sep 09 02:47:07.446293 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.446273 2577 scope.go:117] "RemoveContainer" containerID="7442ee653ddd84f7a574b6052f310bff17a9870febf41692667fa11b542d4b92" Sep 09 02:47:07.446582 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:47:07.446559 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7442ee653ddd84f7a574b6052f310bff17a9870febf41692667fa11b542d4b92\": container with ID starting with 7442ee653ddd84f7a574b6052f310bff17a9870febf41692667fa11b542d4b92 not found: ID does not exist" containerID="7442ee653ddd84f7a574b6052f310bff17a9870febf41692667fa11b542d4b92" Sep 09 02:47:07.446647 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.446595 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7442ee653ddd84f7a574b6052f310bff17a9870febf41692667fa11b542d4b92"} err="failed to get container status \"7442ee653ddd84f7a574b6052f310bff17a9870febf41692667fa11b542d4b92\": rpc error: code = NotFound desc = could not find container \"7442ee653ddd84f7a574b6052f310bff17a9870febf41692667fa11b542d4b92\": container with ID starting with 7442ee653ddd84f7a574b6052f310bff17a9870febf41692667fa11b542d4b92 not found: ID does not exist" Sep 09 02:47:07.459161 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.459139 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv"] Sep 09 02:47:07.461240 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:07.461220 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e17db81-55ff9b79dd-5w9fv"] Sep 09 02:47:08.302318 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:08.302288 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:08.311171 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:08.311144 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e14f6b7f-29ac-43a4-8581-ca0841f70721" path="/var/lib/kubelet/pods/e14f6b7f-29ac-43a4-8581-ca0841f70721/volumes" Sep 09 02:47:08.714330 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:08.714300 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:09.012231 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.012155 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l"] Sep 09 02:47:09.012632 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.012619 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e14f6b7f-29ac-43a4-8581-ca0841f70721" containerName="maas-api" Sep 09 02:47:09.012676 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.012634 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e14f6b7f-29ac-43a4-8581-ca0841f70721" containerName="maas-api" Sep 09 02:47:09.012773 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.012761 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e14f6b7f-29ac-43a4-8581-ca0841f70721" containerName="maas-api" Sep 09 02:47:09.017574 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.017552 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.019996 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.019971 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-1816a546-openshift-default-dockercfg-4xcnv\"" Sep 09 02:47:09.029049 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.029023 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l"] Sep 09 02:47:09.046139 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.046101 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-envoy\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.046298 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.046151 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-workload-socket\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.046298 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.046199 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6wzt\" (UniqueName: \"kubernetes.io/projected/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-kube-api-access-f6wzt\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.046298 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.046232 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-data\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.046298 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.046249 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-podinfo\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.046298 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.046295 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-token\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.046515 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.046319 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-credential-socket\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.046515 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.046353 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-workload-certs\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.046515 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.046376 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istiod-ca-cert\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.048547 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.048515 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:09.147008 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.146972 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-envoy\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.147197 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.147017 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-workload-socket\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.147197 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.147137 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6wzt\" (UniqueName: \"kubernetes.io/projected/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-kube-api-access-f6wzt\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.147322 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.147211 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-data\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.147322 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.147239 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-podinfo\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.147322 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.147318 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-workload-socket\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.147471 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.147359 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-token\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.147471 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.147393 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-credential-socket\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.147471 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.147441 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-workload-certs\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.147638 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.147479 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istiod-ca-cert\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.147638 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.147508 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-data\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.147791 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.147771 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-credential-socket\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.147904 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.147861 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-workload-certs\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.148154 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.148134 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istiod-ca-cert\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.149546 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.149523 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-envoy\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.149655 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.149638 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-podinfo\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.156219 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.156192 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-token\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.156355 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.156338 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6wzt\" (UniqueName: \"kubernetes.io/projected/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-kube-api-access-f6wzt\") pod \"e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.329970 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.329886 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:09.331917 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.331889 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:09.401185 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.401121 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:09.613302 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.613274 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l"] Sep 09 02:47:09.615069 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:47:09.615029 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5d3c27_0cf6_4451_a437_f5d2f4fba41a.slice/crio-7a4efa780a89189c95ea7f09d7ad9e9a14468abb35e7d166c89610184c920dec WatchSource:0}: Error finding container 7a4efa780a89189c95ea7f09d7ad9e9a14468abb35e7d166c89610184c920dec: Status 404 returned error can't find the container with id 7a4efa780a89189c95ea7f09d7ad9e9a14468abb35e7d166c89610184c920dec Sep 09 02:47:09.617387 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.617354 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:47:09.617511 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.617423 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:47:09.617511 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:09.617459 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:47:10.454791 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:10.454761 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" event={"ID":"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a","Type":"ContainerStarted","Data":"d407e64082e62441f1717d784844db840e5d4162413cd0ebf70a1d6845f2ea81"} Sep 09 02:47:10.455166 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:10.454810 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" event={"ID":"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a","Type":"ContainerStarted","Data":"7a4efa780a89189c95ea7f09d7ad9e9a14468abb35e7d166c89610184c920dec"} Sep 09 02:47:10.476722 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:10.476677 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" podStartSLOduration=2.476662571 podStartE2EDuration="2.476662571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:47:10.475142809 +0000 UTC m=+1018.605384907" watchObservedRunningTime="2026-09-09 02:47:10.476662571 +0000 UTC m=+1018.606904667" Sep 09 02:47:11.330089 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:11.330051 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:11.335033 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:11.335010 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:11.460117 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:11.460092 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:11.461026 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:11.461007 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:11.824123 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:11.824091 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:11.951361 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:11.951324 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:12.764887 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:12.764731 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:12.880937 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:12.880900 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:13.715777 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:13.715742 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g"] Sep 09 02:47:13.720610 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:13.720586 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" Sep 09 02:47:13.723198 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:13.723172 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-1816a546-serving-cert\"" Sep 09 02:47:13.733226 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:13.733199 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g"] Sep 09 02:47:13.762244 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:13.762218 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:13.787558 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:13.787530 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-metrics-certs\") pod \"maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g\" (UID: \"8f7b93aa-f8e9-4dc5-9003-cfe75193e969\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" Sep 09 02:47:13.787920 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:13.787570 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cpll\" (UniqueName: \"kubernetes.io/projected/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-kube-api-access-5cpll\") pod \"maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g\" (UID: \"8f7b93aa-f8e9-4dc5-9003-cfe75193e969\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" Sep 09 02:47:13.787920 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:13.787674 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-openshift-service-ca\") pod \"maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g\" (UID: \"8f7b93aa-f8e9-4dc5-9003-cfe75193e969\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" Sep 09 02:47:13.787920 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:13.787736 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-maas-api-tls\") pod \"maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g\" (UID: \"8f7b93aa-f8e9-4dc5-9003-cfe75193e969\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" Sep 09 02:47:13.821186 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:13.821155 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:13.888772 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:13.888736 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-metrics-certs\") pod \"maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g\" (UID: \"8f7b93aa-f8e9-4dc5-9003-cfe75193e969\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" Sep 09 02:47:13.888962 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:13.888783 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cpll\" (UniqueName: \"kubernetes.io/projected/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-kube-api-access-5cpll\") pod \"maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g\" (UID: \"8f7b93aa-f8e9-4dc5-9003-cfe75193e969\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" Sep 09 02:47:13.888962 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:13.888851 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-openshift-service-ca\") pod \"maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g\" (UID: \"8f7b93aa-f8e9-4dc5-9003-cfe75193e969\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" Sep 09 02:47:13.888962 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:13.888920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-maas-api-tls\") pod \"maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g\" (UID: \"8f7b93aa-f8e9-4dc5-9003-cfe75193e969\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" Sep 09 02:47:13.889070 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:47:13.889036 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-serving-cert: secret "maas-api-e2e-derive-1816a546-serving-cert" not found Sep 09 02:47:13.889110 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:47:13.889099 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-maas-api-tls podName:8f7b93aa-f8e9-4dc5-9003-cfe75193e969 nodeName:}" failed. No retries permitted until 2026-09-09 02:47:14.389079569 +0000 UTC m=+1022.519321668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-maas-api-tls") pod "maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" (UID: "8f7b93aa-f8e9-4dc5-9003-cfe75193e969") : secret "maas-api-e2e-derive-1816a546-serving-cert" not found Sep 09 02:47:13.889554 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:13.889530 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-openshift-service-ca\") pod \"maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g\" (UID: \"8f7b93aa-f8e9-4dc5-9003-cfe75193e969\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" Sep 09 02:47:13.891293 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:13.891267 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-metrics-certs\") pod \"maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g\" (UID: \"8f7b93aa-f8e9-4dc5-9003-cfe75193e969\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" Sep 09 02:47:13.897965 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:13.897940 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cpll\" (UniqueName: \"kubernetes.io/projected/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-kube-api-access-5cpll\") pod \"maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g\" (UID: \"8f7b93aa-f8e9-4dc5-9003-cfe75193e969\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" Sep 09 02:47:14.203527 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:14.203493 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:14.305120 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:14.305088 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:14.393922 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:14.393879 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-maas-api-tls\") pod \"maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g\" (UID: \"8f7b93aa-f8e9-4dc5-9003-cfe75193e969\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" Sep 09 02:47:14.396253 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:14.396234 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-maas-api-tls\") pod \"maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g\" (UID: \"8f7b93aa-f8e9-4dc5-9003-cfe75193e969\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" Sep 09 02:47:14.632785 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:14.632750 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" Sep 09 02:47:14.810667 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:14.810636 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:14.912752 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:14.912725 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g"] Sep 09 02:47:14.915537 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:47:14.915503 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7b93aa_f8e9_4dc5_9003_cfe75193e969.slice/crio-e1f9626b43b3ebd4eb57d91dda0cf1a0a0f0a04044296a5058d4a5e4b96c22c7 WatchSource:0}: Error finding container e1f9626b43b3ebd4eb57d91dda0cf1a0a0f0a04044296a5058d4a5e4b96c22c7: Status 404 returned error can't find the container with id e1f9626b43b3ebd4eb57d91dda0cf1a0a0f0a04044296a5058d4a5e4b96c22c7 Sep 09 02:47:14.917576 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:14.917543 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 02:47:14.953555 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:14.953500 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:15.482516 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:15.482481 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" event={"ID":"8f7b93aa-f8e9-4dc5-9003-cfe75193e969","Type":"ContainerStarted","Data":"a4626c03199c654868b1ceffcbb58345c85419beea61782c6a448310ec095b56"} Sep 09 02:47:15.482681 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:15.482551 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" event={"ID":"8f7b93aa-f8e9-4dc5-9003-cfe75193e969","Type":"ContainerStarted","Data":"e1f9626b43b3ebd4eb57d91dda0cf1a0a0f0a04044296a5058d4a5e4b96c22c7"} Sep 09 02:47:15.482733 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:15.482714 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" Sep 09 02:47:15.501268 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:15.501223 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" podStartSLOduration=2.082581595 podStartE2EDuration="2.501208983s" totalImagesPullingTime="418.627388ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:47:15.501182374 +0000 UTC m=+1023.631424464" watchObservedRunningTime="2026-09-09 02:47:15.501208983 +0000 UTC m=+1023.631451073" Sep 09 02:47:15.547289 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:15.547261 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:21.494591 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:21.494560 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" Sep 09 02:47:22.859296 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:22.859262 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:23.552856 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:23.552801 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4"] Sep 09 02:47:23.553129 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:23.553096 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" podUID="99717fb1-9461-43dd-81bd-fc113435e140" containerName="maas-api" containerID="cri-o://9ca2f6b01e00b45285d308824d7e38db6d2c9103f8e1d365553a5cca097888f8" gracePeriod=30 Sep 09 02:47:23.581253 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:23.581225 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:23.652712 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:23.652677 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:23.752021 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:23.751983 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:23.803843 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:23.803775 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" Sep 09 02:47:23.826280 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:23.826255 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:23.900771 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:23.900740 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99717fb1-9461-43dd-81bd-fc113435e140-openshift-service-ca\") pod \"99717fb1-9461-43dd-81bd-fc113435e140\" (UID: \"99717fb1-9461-43dd-81bd-fc113435e140\") " Sep 09 02:47:23.901269 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:23.900810 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fwnf\" (UniqueName: \"kubernetes.io/projected/99717fb1-9461-43dd-81bd-fc113435e140-kube-api-access-9fwnf\") pod \"99717fb1-9461-43dd-81bd-fc113435e140\" (UID: \"99717fb1-9461-43dd-81bd-fc113435e140\") " Sep 09 02:47:23.901269 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:23.900888 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99717fb1-9461-43dd-81bd-fc113435e140-maas-api-tls\") pod \"99717fb1-9461-43dd-81bd-fc113435e140\" (UID: \"99717fb1-9461-43dd-81bd-fc113435e140\") " Sep 09 02:47:23.901269 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:23.900952 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99717fb1-9461-43dd-81bd-fc113435e140-metrics-certs\") pod \"99717fb1-9461-43dd-81bd-fc113435e140\" (UID: \"99717fb1-9461-43dd-81bd-fc113435e140\") " Sep 09 02:47:23.901269 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:23.901155 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99717fb1-9461-43dd-81bd-fc113435e140-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "99717fb1-9461-43dd-81bd-fc113435e140" (UID: "99717fb1-9461-43dd-81bd-fc113435e140"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:47:23.901513 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:23.901309 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99717fb1-9461-43dd-81bd-fc113435e140-openshift-service-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:23.903060 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:23.903034 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99717fb1-9461-43dd-81bd-fc113435e140-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "99717fb1-9461-43dd-81bd-fc113435e140" (UID: "99717fb1-9461-43dd-81bd-fc113435e140"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:47:23.903463 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:23.903437 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99717fb1-9461-43dd-81bd-fc113435e140-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "99717fb1-9461-43dd-81bd-fc113435e140" (UID: "99717fb1-9461-43dd-81bd-fc113435e140"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:47:23.903564 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:23.903505 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99717fb1-9461-43dd-81bd-fc113435e140-kube-api-access-9fwnf" (OuterVolumeSpecName: "kube-api-access-9fwnf") pod "99717fb1-9461-43dd-81bd-fc113435e140" (UID: "99717fb1-9461-43dd-81bd-fc113435e140"). InnerVolumeSpecName "kube-api-access-9fwnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:47:24.002400 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:24.002365 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99717fb1-9461-43dd-81bd-fc113435e140-metrics-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:24.002400 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:24.002394 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fwnf\" (UniqueName: \"kubernetes.io/projected/99717fb1-9461-43dd-81bd-fc113435e140-kube-api-access-9fwnf\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:24.002400 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:24.002403 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99717fb1-9461-43dd-81bd-fc113435e140-maas-api-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:24.173090 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:24.173049 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:24.534460 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:24.534367 2577 generic.go:356] "Generic (PLEG): container finished" podID="99717fb1-9461-43dd-81bd-fc113435e140" containerID="9ca2f6b01e00b45285d308824d7e38db6d2c9103f8e1d365553a5cca097888f8" exitCode=0 Sep 09 02:47:24.534460 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:24.534435 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" Sep 09 02:47:24.534678 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:24.534466 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" event={"ID":"99717fb1-9461-43dd-81bd-fc113435e140","Type":"ContainerDied","Data":"9ca2f6b01e00b45285d308824d7e38db6d2c9103f8e1d365553a5cca097888f8"} Sep 09 02:47:24.534678 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:24.534532 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4" event={"ID":"99717fb1-9461-43dd-81bd-fc113435e140","Type":"ContainerDied","Data":"c7295cfc04f688e934ad652a406689ac8c69c822e905e2340bbce08432a8b6d6"} Sep 09 02:47:24.534678 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:24.534572 2577 scope.go:117] "RemoveContainer" containerID="9ca2f6b01e00b45285d308824d7e38db6d2c9103f8e1d365553a5cca097888f8" Sep 09 02:47:24.544589 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:24.544569 2577 scope.go:117] "RemoveContainer" containerID="9ca2f6b01e00b45285d308824d7e38db6d2c9103f8e1d365553a5cca097888f8" Sep 09 02:47:24.544911 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:47:24.544882 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca2f6b01e00b45285d308824d7e38db6d2c9103f8e1d365553a5cca097888f8\": container with ID starting with 9ca2f6b01e00b45285d308824d7e38db6d2c9103f8e1d365553a5cca097888f8 not found: ID does not exist" containerID="9ca2f6b01e00b45285d308824d7e38db6d2c9103f8e1d365553a5cca097888f8" Sep 09 02:47:24.545009 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:24.544921 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca2f6b01e00b45285d308824d7e38db6d2c9103f8e1d365553a5cca097888f8"} err="failed to get container status \"9ca2f6b01e00b45285d308824d7e38db6d2c9103f8e1d365553a5cca097888f8\": rpc error: code = NotFound desc = could not find container \"9ca2f6b01e00b45285d308824d7e38db6d2c9103f8e1d365553a5cca097888f8\": container with ID starting with 9ca2f6b01e00b45285d308824d7e38db6d2c9103f8e1d365553a5cca097888f8 not found: ID does not exist" Sep 09 02:47:24.554679 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:24.554627 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4"] Sep 09 02:47:24.560467 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:24.560431 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-07b2d4-7588c9c6df-kqtw4"] Sep 09 02:47:26.311155 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:26.311120 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99717fb1-9461-43dd-81bd-fc113435e140" path="/var/lib/kubelet/pods/99717fb1-9461-43dd-81bd-fc113435e140/volumes" Sep 09 02:47:29.253085 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:29.253030 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:33.593443 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:33.593409 2577 generic.go:356] "Generic (PLEG): container finished" podID="8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" containerID="d407e64082e62441f1717d784844db840e5d4162413cd0ebf70a1d6845f2ea81" exitCode=0 Sep 09 02:47:33.593920 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:33.593493 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" event={"ID":"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a","Type":"ContainerDied","Data":"d407e64082e62441f1717d784844db840e5d4162413cd0ebf70a1d6845f2ea81"} Sep 09 02:47:33.593972 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:33.593950 2577 scope.go:117] "RemoveContainer" containerID="d407e64082e62441f1717d784844db840e5d4162413cd0ebf70a1d6845f2ea81" Sep 09 02:47:33.595012 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:33.594987 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:47:33.595082 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:33.595055 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:47:33.595120 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:33.595088 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:47:34.600897 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:34.600867 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" event={"ID":"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a","Type":"ContainerStarted","Data":"da4ecfcfedb18af3f9f5fd59d1f8e23d5103f2b2fa741b9aca6d9f1280c8af15"} Sep 09 02:47:34.629876 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:34.629849 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:35.330753 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:35.330713 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:35.330980 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:35.330916 2577 patch_prober.go:28] interesting pod/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 09 02:47:35.331056 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:35.330980 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" podUID="8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 09 02:47:35.641641 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:35.641611 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:35.890073 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:35.890036 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj"] Sep 09 02:47:35.890378 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:35.890326 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" podUID="0b5deaea-bded-4181-99aa-69d4a867daaa" containerName="istio-proxy" containerID="cri-o://3b8dbc91e35076b90e6b9fb1baac506f4cea8fcbb779b921fc9d1c2726221b67" gracePeriod=30 Sep 09 02:47:35.898165 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:35.898086 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:36.331178 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:36.331078 2577 patch_prober.go:28] interesting pod/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 09 02:47:36.331178 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:36.331164 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" podUID="8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 09 02:47:36.372354 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:36.372286 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:36.730906 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:36.730868 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:36.996623 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:36.996539 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:37.330874 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:37.330792 2577 patch_prober.go:28] interesting pod/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 09 02:47:37.331023 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:37.330872 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" podUID="8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 09 02:47:38.330666 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:38.330634 2577 patch_prober.go:28] interesting pod/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 09 02:47:38.331102 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:38.330687 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" podUID="8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 09 02:47:39.330612 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:39.330569 2577 patch_prober.go:28] interesting pod/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 09 02:47:39.330856 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:39.330650 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" podUID="8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 09 02:47:39.886561 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:39.886516 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:39.976311 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:39.976271 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:40.094288 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:40.094251 2577 patch_prober.go:28] interesting pod/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 09 02:47:40.094467 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:40.094321 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" podUID="0b5deaea-bded-4181-99aa-69d4a867daaa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 02:47:40.335354 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:40.335274 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:40.335727 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:40.335643 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:40.336226 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:40.336210 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:41.023527 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.023491 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695"] Sep 09 02:47:41.024247 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.024224 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99717fb1-9461-43dd-81bd-fc113435e140" containerName="maas-api" Sep 09 02:47:41.024345 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.024251 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="99717fb1-9461-43dd-81bd-fc113435e140" containerName="maas-api" Sep 09 02:47:41.024413 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.024401 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="99717fb1-9461-43dd-81bd-fc113435e140" containerName="maas-api" Sep 09 02:47:41.026943 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.026922 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" Sep 09 02:47:41.029753 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.029727 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-b52336-serving-cert\"" Sep 09 02:47:41.039729 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.039665 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695"] Sep 09 02:47:41.064785 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.064733 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:41.161759 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.161671 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:41.161945 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.161877 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:47:41.178796 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.178767 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86z5d\" (UniqueName: \"kubernetes.io/projected/a981159c-f4ba-4d23-b4e7-54bec16872ee-kube-api-access-86z5d\") pod \"maas-api-e2e-explicit-b52336-77df5c888-87695\" (UID: \"a981159c-f4ba-4d23-b4e7-54bec16872ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" Sep 09 02:47:41.178935 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.178804 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a981159c-f4ba-4d23-b4e7-54bec16872ee-metrics-certs\") pod \"maas-api-e2e-explicit-b52336-77df5c888-87695\" (UID: \"a981159c-f4ba-4d23-b4e7-54bec16872ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" Sep 09 02:47:41.178935 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.178839 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a981159c-f4ba-4d23-b4e7-54bec16872ee-maas-api-tls\") pod \"maas-api-e2e-explicit-b52336-77df5c888-87695\" (UID: \"a981159c-f4ba-4d23-b4e7-54bec16872ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" Sep 09 02:47:41.178935 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.178860 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a981159c-f4ba-4d23-b4e7-54bec16872ee-openshift-service-ca\") pod \"maas-api-e2e-explicit-b52336-77df5c888-87695\" (UID: \"a981159c-f4ba-4d23-b4e7-54bec16872ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" Sep 09 02:47:41.259640 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.259606 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:41.279781 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.279760 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-data\") pod \"0b5deaea-bded-4181-99aa-69d4a867daaa\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " Sep 09 02:47:41.279929 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.279796 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-envoy\") pod \"0b5deaea-bded-4181-99aa-69d4a867daaa\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " Sep 09 02:47:41.279929 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.279837 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-credential-socket\") pod \"0b5deaea-bded-4181-99aa-69d4a867daaa\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " Sep 09 02:47:41.279929 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.279872 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cfmz\" (UniqueName: \"kubernetes.io/projected/0b5deaea-bded-4181-99aa-69d4a867daaa-kube-api-access-7cfmz\") pod \"0b5deaea-bded-4181-99aa-69d4a867daaa\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " Sep 09 02:47:41.279929 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.279923 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-token\") pod \"0b5deaea-bded-4181-99aa-69d4a867daaa\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " Sep 09 02:47:41.280210 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.279959 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-podinfo\") pod \"0b5deaea-bded-4181-99aa-69d4a867daaa\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " Sep 09 02:47:41.280210 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.280009 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5deaea-bded-4181-99aa-69d4a867daaa-istiod-ca-cert\") pod \"0b5deaea-bded-4181-99aa-69d4a867daaa\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " Sep 09 02:47:41.280464 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.280032 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-workload-certs\") pod \"0b5deaea-bded-4181-99aa-69d4a867daaa\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " Sep 09 02:47:41.280548 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.280532 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-workload-socket\") pod \"0b5deaea-bded-4181-99aa-69d4a867daaa\" (UID: \"0b5deaea-bded-4181-99aa-69d4a867daaa\") " Sep 09 02:47:41.280611 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.280115 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-data" (OuterVolumeSpecName: "istio-data") pod "0b5deaea-bded-4181-99aa-69d4a867daaa" (UID: "0b5deaea-bded-4181-99aa-69d4a867daaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:47:41.280611 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.280198 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0b5deaea-bded-4181-99aa-69d4a867daaa" (UID: "0b5deaea-bded-4181-99aa-69d4a867daaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:47:41.280611 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.280337 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0b5deaea-bded-4181-99aa-69d4a867daaa" (UID: "0b5deaea-bded-4181-99aa-69d4a867daaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:47:41.280611 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.280424 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b5deaea-bded-4181-99aa-69d4a867daaa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0b5deaea-bded-4181-99aa-69d4a867daaa" (UID: "0b5deaea-bded-4181-99aa-69d4a867daaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:47:41.280835 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.280761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86z5d\" (UniqueName: \"kubernetes.io/projected/a981159c-f4ba-4d23-b4e7-54bec16872ee-kube-api-access-86z5d\") pod \"maas-api-e2e-explicit-b52336-77df5c888-87695\" (UID: \"a981159c-f4ba-4d23-b4e7-54bec16872ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" Sep 09 02:47:41.280930 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.280906 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0b5deaea-bded-4181-99aa-69d4a867daaa" (UID: "0b5deaea-bded-4181-99aa-69d4a867daaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:47:41.280991 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.280951 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a981159c-f4ba-4d23-b4e7-54bec16872ee-metrics-certs\") pod \"maas-api-e2e-explicit-b52336-77df5c888-87695\" (UID: \"a981159c-f4ba-4d23-b4e7-54bec16872ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" Sep 09 02:47:41.281062 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.281002 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a981159c-f4ba-4d23-b4e7-54bec16872ee-maas-api-tls\") pod \"maas-api-e2e-explicit-b52336-77df5c888-87695\" (UID: \"a981159c-f4ba-4d23-b4e7-54bec16872ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" Sep 09 02:47:41.281062 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.281036 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a981159c-f4ba-4d23-b4e7-54bec16872ee-openshift-service-ca\") pod \"maas-api-e2e-explicit-b52336-77df5c888-87695\" (UID: \"a981159c-f4ba-4d23-b4e7-54bec16872ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" Sep 09 02:47:41.281272 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.281245 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-workload-socket\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:41.281272 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.281269 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-data\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:41.281419 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.281284 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-credential-socket\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:41.281419 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.281298 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5deaea-bded-4181-99aa-69d4a867daaa-istiod-ca-cert\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:41.281419 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.281310 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-workload-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:41.281890 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.281863 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a981159c-f4ba-4d23-b4e7-54bec16872ee-openshift-service-ca\") pod \"maas-api-e2e-explicit-b52336-77df5c888-87695\" (UID: \"a981159c-f4ba-4d23-b4e7-54bec16872ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" Sep 09 02:47:41.282730 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.282674 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0b5deaea-bded-4181-99aa-69d4a867daaa" (UID: "0b5deaea-bded-4181-99aa-69d4a867daaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 02:47:41.282730 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.282723 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5deaea-bded-4181-99aa-69d4a867daaa-kube-api-access-7cfmz" (OuterVolumeSpecName: "kube-api-access-7cfmz") pod "0b5deaea-bded-4181-99aa-69d4a867daaa" (UID: "0b5deaea-bded-4181-99aa-69d4a867daaa"). InnerVolumeSpecName "kube-api-access-7cfmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:47:41.282895 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.282750 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0b5deaea-bded-4181-99aa-69d4a867daaa" (UID: "0b5deaea-bded-4181-99aa-69d4a867daaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:47:41.282895 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.282787 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b5deaea-bded-4181-99aa-69d4a867daaa" (UID: "0b5deaea-bded-4181-99aa-69d4a867daaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:47:41.283661 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.283636 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a981159c-f4ba-4d23-b4e7-54bec16872ee-metrics-certs\") pod \"maas-api-e2e-explicit-b52336-77df5c888-87695\" (UID: \"a981159c-f4ba-4d23-b4e7-54bec16872ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" Sep 09 02:47:41.283841 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.283800 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a981159c-f4ba-4d23-b4e7-54bec16872ee-maas-api-tls\") pod \"maas-api-e2e-explicit-b52336-77df5c888-87695\" (UID: \"a981159c-f4ba-4d23-b4e7-54bec16872ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" Sep 09 02:47:41.289863 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.289843 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86z5d\" (UniqueName: \"kubernetes.io/projected/a981159c-f4ba-4d23-b4e7-54bec16872ee-kube-api-access-86z5d\") pod \"maas-api-e2e-explicit-b52336-77df5c888-87695\" (UID: \"a981159c-f4ba-4d23-b4e7-54bec16872ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" Sep 09 02:47:41.341752 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.341717 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" Sep 09 02:47:41.382299 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.382269 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-podinfo\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:41.382299 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.382294 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-envoy\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:41.382299 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.382304 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cfmz\" (UniqueName: \"kubernetes.io/projected/0b5deaea-bded-4181-99aa-69d4a867daaa-kube-api-access-7cfmz\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:41.382515 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.382313 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5deaea-bded-4181-99aa-69d4a867daaa-istio-token\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:41.612505 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.612472 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695"] Sep 09 02:47:41.614638 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:47:41.614609 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda981159c_f4ba_4d23_b4e7_54bec16872ee.slice/crio-ddb6aa0261f54ee87569f1781acb3fccffd9988990091d1ad2f2c98f62bcc63a WatchSource:0}: Error finding container ddb6aa0261f54ee87569f1781acb3fccffd9988990091d1ad2f2c98f62bcc63a: Status 404 returned error can't find the container with id ddb6aa0261f54ee87569f1781acb3fccffd9988990091d1ad2f2c98f62bcc63a Sep 09 02:47:41.644239 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.644210 2577 generic.go:356] "Generic (PLEG): container finished" podID="0b5deaea-bded-4181-99aa-69d4a867daaa" containerID="3b8dbc91e35076b90e6b9fb1baac506f4cea8fcbb779b921fc9d1c2726221b67" exitCode=0 Sep 09 02:47:41.644428 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.644274 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" Sep 09 02:47:41.644428 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.644293 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" event={"ID":"0b5deaea-bded-4181-99aa-69d4a867daaa","Type":"ContainerDied","Data":"3b8dbc91e35076b90e6b9fb1baac506f4cea8fcbb779b921fc9d1c2726221b67"} Sep 09 02:47:41.644428 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.644341 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj" event={"ID":"0b5deaea-bded-4181-99aa-69d4a867daaa","Type":"ContainerDied","Data":"82959b5aff22adaa4f35851abe07f075fdbd7b17bb034965ec264cbcdc0f4218"} Sep 09 02:47:41.644978 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.644373 2577 scope.go:117] "RemoveContainer" containerID="3b8dbc91e35076b90e6b9fb1baac506f4cea8fcbb779b921fc9d1c2726221b67" Sep 09 02:47:41.645863 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.645779 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" event={"ID":"a981159c-f4ba-4d23-b4e7-54bec16872ee","Type":"ContainerStarted","Data":"ddb6aa0261f54ee87569f1781acb3fccffd9988990091d1ad2f2c98f62bcc63a"} Sep 09 02:47:41.654230 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.654210 2577 scope.go:117] "RemoveContainer" containerID="2e7d71b73df0a64a36efd486f29a2605e3f63c8ecb4ae9743e7b0b113db4bb3a" Sep 09 02:47:41.662409 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.662389 2577 scope.go:117] "RemoveContainer" containerID="3b8dbc91e35076b90e6b9fb1baac506f4cea8fcbb779b921fc9d1c2726221b67" Sep 09 02:47:41.662717 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:47:41.662699 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b8dbc91e35076b90e6b9fb1baac506f4cea8fcbb779b921fc9d1c2726221b67\": container with ID starting with 3b8dbc91e35076b90e6b9fb1baac506f4cea8fcbb779b921fc9d1c2726221b67 not found: ID does not exist" containerID="3b8dbc91e35076b90e6b9fb1baac506f4cea8fcbb779b921fc9d1c2726221b67" Sep 09 02:47:41.662780 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.662725 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b8dbc91e35076b90e6b9fb1baac506f4cea8fcbb779b921fc9d1c2726221b67"} err="failed to get container status \"3b8dbc91e35076b90e6b9fb1baac506f4cea8fcbb779b921fc9d1c2726221b67\": rpc error: code = NotFound desc = could not find container \"3b8dbc91e35076b90e6b9fb1baac506f4cea8fcbb779b921fc9d1c2726221b67\": container with ID starting with 3b8dbc91e35076b90e6b9fb1baac506f4cea8fcbb779b921fc9d1c2726221b67 not found: ID does not exist" Sep 09 02:47:41.662780 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.662741 2577 scope.go:117] "RemoveContainer" containerID="2e7d71b73df0a64a36efd486f29a2605e3f63c8ecb4ae9743e7b0b113db4bb3a" Sep 09 02:47:41.663036 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:47:41.663019 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7d71b73df0a64a36efd486f29a2605e3f63c8ecb4ae9743e7b0b113db4bb3a\": container with ID starting with 2e7d71b73df0a64a36efd486f29a2605e3f63c8ecb4ae9743e7b0b113db4bb3a not found: ID does not exist" containerID="2e7d71b73df0a64a36efd486f29a2605e3f63c8ecb4ae9743e7b0b113db4bb3a" Sep 09 02:47:41.663082 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.663040 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7d71b73df0a64a36efd486f29a2605e3f63c8ecb4ae9743e7b0b113db4bb3a"} err="failed to get container status \"2e7d71b73df0a64a36efd486f29a2605e3f63c8ecb4ae9743e7b0b113db4bb3a\": rpc error: code = NotFound desc = could not find container \"2e7d71b73df0a64a36efd486f29a2605e3f63c8ecb4ae9743e7b0b113db4bb3a\": container with ID starting with 2e7d71b73df0a64a36efd486f29a2605e3f63c8ecb4ae9743e7b0b113db4bb3a not found: ID does not exist" Sep 09 02:47:41.669418 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.669398 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj"] Sep 09 02:47:41.672084 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.672065 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:41.675550 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.675530 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-07b2d4-openshift-default-5b845c8b48-p2bwj"] Sep 09 02:47:41.768970 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:41.768945 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:42.311651 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:42.311561 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b5deaea-bded-4181-99aa-69d4a867daaa" path="/var/lib/kubelet/pods/0b5deaea-bded-4181-99aa-69d4a867daaa/volumes" Sep 09 02:47:42.653827 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:42.653790 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" event={"ID":"a981159c-f4ba-4d23-b4e7-54bec16872ee","Type":"ContainerStarted","Data":"7d6f411fe6723fa536f84947523a85a2e8869e345f6dbe13cf0ab4fad5209c4d"} Sep 09 02:47:42.654238 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:42.654029 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" Sep 09 02:47:42.673080 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:42.673030 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" podStartSLOduration=2.335531546 podStartE2EDuration="2.67301425s" totalImagesPullingTime="337.482704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:47:42.671261675 +0000 UTC m=+1050.801503771" watchObservedRunningTime="2026-09-09 02:47:42.67301425 +0000 UTC m=+1050.803256346" Sep 09 02:47:43.263452 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.263417 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g"] Sep 09 02:47:43.263660 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.263631 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" podUID="8f7b93aa-f8e9-4dc5-9003-cfe75193e969" containerName="maas-api" containerID="cri-o://a4626c03199c654868b1ceffcbb58345c85419beea61782c6a448310ec095b56" gracePeriod=30 Sep 09 02:47:43.293355 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.293330 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:43.371135 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.371094 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:43.432927 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.432835 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:43.506132 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.506106 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:43.510122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.510103 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" Sep 09 02:47:43.564520 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.564434 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:43.605609 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.605577 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cpll\" (UniqueName: \"kubernetes.io/projected/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-kube-api-access-5cpll\") pod \"8f7b93aa-f8e9-4dc5-9003-cfe75193e969\" (UID: \"8f7b93aa-f8e9-4dc5-9003-cfe75193e969\") " Sep 09 02:47:43.605747 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.605694 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-maas-api-tls\") pod \"8f7b93aa-f8e9-4dc5-9003-cfe75193e969\" (UID: \"8f7b93aa-f8e9-4dc5-9003-cfe75193e969\") " Sep 09 02:47:43.605747 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.605738 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-openshift-service-ca\") pod \"8f7b93aa-f8e9-4dc5-9003-cfe75193e969\" (UID: \"8f7b93aa-f8e9-4dc5-9003-cfe75193e969\") " Sep 09 02:47:43.605895 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.605771 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-metrics-certs\") pod \"8f7b93aa-f8e9-4dc5-9003-cfe75193e969\" (UID: \"8f7b93aa-f8e9-4dc5-9003-cfe75193e969\") " Sep 09 02:47:43.606150 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.606118 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8f7b93aa-f8e9-4dc5-9003-cfe75193e969" (UID: "8f7b93aa-f8e9-4dc5-9003-cfe75193e969"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:47:43.606239 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.606222 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-openshift-service-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:43.607753 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.607731 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-kube-api-access-5cpll" (OuterVolumeSpecName: "kube-api-access-5cpll") pod "8f7b93aa-f8e9-4dc5-9003-cfe75193e969" (UID: "8f7b93aa-f8e9-4dc5-9003-cfe75193e969"). InnerVolumeSpecName "kube-api-access-5cpll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:47:43.608019 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.608000 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8f7b93aa-f8e9-4dc5-9003-cfe75193e969" (UID: "8f7b93aa-f8e9-4dc5-9003-cfe75193e969"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:47:43.608096 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.608028 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8f7b93aa-f8e9-4dc5-9003-cfe75193e969" (UID: "8f7b93aa-f8e9-4dc5-9003-cfe75193e969"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:47:43.656543 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.656513 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l"] Sep 09 02:47:43.656986 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.656704 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" podUID="8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" containerName="istio-proxy" containerID="cri-o://da4ecfcfedb18af3f9f5fd59d1f8e23d5103f2b2fa741b9aca6d9f1280c8af15" gracePeriod=30 Sep 09 02:47:43.661672 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.661647 2577 generic.go:356] "Generic (PLEG): container finished" podID="8f7b93aa-f8e9-4dc5-9003-cfe75193e969" containerID="a4626c03199c654868b1ceffcbb58345c85419beea61782c6a448310ec095b56" exitCode=0 Sep 09 02:47:43.661803 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.661683 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" Sep 09 02:47:43.661803 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.661723 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" event={"ID":"8f7b93aa-f8e9-4dc5-9003-cfe75193e969","Type":"ContainerDied","Data":"a4626c03199c654868b1ceffcbb58345c85419beea61782c6a448310ec095b56"} Sep 09 02:47:43.661803 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.661770 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g" event={"ID":"8f7b93aa-f8e9-4dc5-9003-cfe75193e969","Type":"ContainerDied","Data":"e1f9626b43b3ebd4eb57d91dda0cf1a0a0f0a04044296a5058d4a5e4b96c22c7"} Sep 09 02:47:43.661803 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.661792 2577 scope.go:117] "RemoveContainer" containerID="a4626c03199c654868b1ceffcbb58345c85419beea61782c6a448310ec095b56" Sep 09 02:47:43.671709 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.671683 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:43.671934 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.671872 2577 scope.go:117] "RemoveContainer" containerID="a4626c03199c654868b1ceffcbb58345c85419beea61782c6a448310ec095b56" Sep 09 02:47:43.672278 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:47:43.672249 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4626c03199c654868b1ceffcbb58345c85419beea61782c6a448310ec095b56\": container with ID starting with a4626c03199c654868b1ceffcbb58345c85419beea61782c6a448310ec095b56 not found: ID does not exist" containerID="a4626c03199c654868b1ceffcbb58345c85419beea61782c6a448310ec095b56" Sep 09 02:47:43.672359 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.672276 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4626c03199c654868b1ceffcbb58345c85419beea61782c6a448310ec095b56"} err="failed to get container status \"a4626c03199c654868b1ceffcbb58345c85419beea61782c6a448310ec095b56\": rpc error: code = NotFound desc = could not find container \"a4626c03199c654868b1ceffcbb58345c85419beea61782c6a448310ec095b56\": container with ID starting with a4626c03199c654868b1ceffcbb58345c85419beea61782c6a448310ec095b56 not found: ID does not exist" Sep 09 02:47:43.694373 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.694344 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g"] Sep 09 02:47:43.698289 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.698269 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1816a546-644bcdbb96-8bl7g"] Sep 09 02:47:43.707453 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.707429 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-metrics-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:43.707556 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.707456 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cpll\" (UniqueName: \"kubernetes.io/projected/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-kube-api-access-5cpll\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:43.707556 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:43.707471 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f7b93aa-f8e9-4dc5-9003-cfe75193e969-maas-api-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:44.310594 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:44.310562 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f7b93aa-f8e9-4dc5-9003-cfe75193e969" path="/var/lib/kubelet/pods/8f7b93aa-f8e9-4dc5-9003-cfe75193e969/volumes" Sep 09 02:47:44.384228 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:44.384186 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:44.480520 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:44.480492 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:44.580935 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:44.580847 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:44.930009 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:44.929981 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:45.981310 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:45.981259 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:48.667684 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:48.667653 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" Sep 09 02:47:48.908171 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:48.908148 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:49.060533 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.060442 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-envoy\") pod \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " Sep 09 02:47:49.060533 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.060515 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istiod-ca-cert\") pod \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " Sep 09 02:47:49.060797 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.060541 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-workload-socket\") pod \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " Sep 09 02:47:49.060797 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.060563 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-token\") pod \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " Sep 09 02:47:49.060797 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.060591 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6wzt\" (UniqueName: \"kubernetes.io/projected/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-kube-api-access-f6wzt\") pod \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " Sep 09 02:47:49.060797 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.060638 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-data\") pod \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " Sep 09 02:47:49.060797 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.060684 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-credential-socket\") pod \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " Sep 09 02:47:49.060797 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.060736 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-workload-certs\") pod \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " Sep 09 02:47:49.060797 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.060795 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-podinfo\") pod \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\" (UID: \"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a\") " Sep 09 02:47:49.061203 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.060916 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" (UID: "8c5d3c27-0cf6-4451-a437-f5d2f4fba41a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:47:49.061203 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.061016 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" (UID: "8c5d3c27-0cf6-4451-a437-f5d2f4fba41a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:47:49.061203 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.061137 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" (UID: "8c5d3c27-0cf6-4451-a437-f5d2f4fba41a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:47:49.061203 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.061189 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-data" (OuterVolumeSpecName: "istio-data") pod "8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" (UID: "8c5d3c27-0cf6-4451-a437-f5d2f4fba41a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:47:49.061203 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.061198 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istiod-ca-cert\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:49.061466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.061224 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-workload-socket\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:49.061466 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.061267 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" (UID: "8c5d3c27-0cf6-4451-a437-f5d2f4fba41a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:47:49.062929 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.062895 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" (UID: "8c5d3c27-0cf6-4451-a437-f5d2f4fba41a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 02:47:49.062929 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.062903 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-token" (OuterVolumeSpecName: "istio-token") pod "8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" (UID: "8c5d3c27-0cf6-4451-a437-f5d2f4fba41a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:47:49.063203 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.063179 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" (UID: "8c5d3c27-0cf6-4451-a437-f5d2f4fba41a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:47:49.063290 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.063204 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-kube-api-access-f6wzt" (OuterVolumeSpecName: "kube-api-access-f6wzt") pod "8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" (UID: "8c5d3c27-0cf6-4451-a437-f5d2f4fba41a"). InnerVolumeSpecName "kube-api-access-f6wzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:47:49.162522 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.162476 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-envoy\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:49.162522 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.162517 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-token\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:49.162522 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.162526 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6wzt\" (UniqueName: \"kubernetes.io/projected/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-kube-api-access-f6wzt\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:49.162522 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.162535 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-data\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:49.162775 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.162544 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-credential-socket\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:49.162775 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.162562 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-workload-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:49.162775 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.162572 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a-istio-podinfo\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:47:49.699506 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.699473 2577 generic.go:356] "Generic (PLEG): container finished" podID="8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" containerID="da4ecfcfedb18af3f9f5fd59d1f8e23d5103f2b2fa741b9aca6d9f1280c8af15" exitCode=0 Sep 09 02:47:49.700051 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.699521 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" Sep 09 02:47:49.700051 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.699552 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" event={"ID":"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a","Type":"ContainerDied","Data":"da4ecfcfedb18af3f9f5fd59d1f8e23d5103f2b2fa741b9aca6d9f1280c8af15"} Sep 09 02:47:49.700051 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.699605 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l" event={"ID":"8c5d3c27-0cf6-4451-a437-f5d2f4fba41a","Type":"ContainerDied","Data":"7a4efa780a89189c95ea7f09d7ad9e9a14468abb35e7d166c89610184c920dec"} Sep 09 02:47:49.700051 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.699628 2577 scope.go:117] "RemoveContainer" containerID="da4ecfcfedb18af3f9f5fd59d1f8e23d5103f2b2fa741b9aca6d9f1280c8af15" Sep 09 02:47:49.709071 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.709052 2577 scope.go:117] "RemoveContainer" containerID="d407e64082e62441f1717d784844db840e5d4162413cd0ebf70a1d6845f2ea81" Sep 09 02:47:49.717915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.717895 2577 scope.go:117] "RemoveContainer" containerID="da4ecfcfedb18af3f9f5fd59d1f8e23d5103f2b2fa741b9aca6d9f1280c8af15" Sep 09 02:47:49.718189 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:47:49.718169 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da4ecfcfedb18af3f9f5fd59d1f8e23d5103f2b2fa741b9aca6d9f1280c8af15\": container with ID starting with da4ecfcfedb18af3f9f5fd59d1f8e23d5103f2b2fa741b9aca6d9f1280c8af15 not found: ID does not exist" containerID="da4ecfcfedb18af3f9f5fd59d1f8e23d5103f2b2fa741b9aca6d9f1280c8af15" Sep 09 02:47:49.718230 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.718198 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da4ecfcfedb18af3f9f5fd59d1f8e23d5103f2b2fa741b9aca6d9f1280c8af15"} err="failed to get container status \"da4ecfcfedb18af3f9f5fd59d1f8e23d5103f2b2fa741b9aca6d9f1280c8af15\": rpc error: code = NotFound desc = could not find container \"da4ecfcfedb18af3f9f5fd59d1f8e23d5103f2b2fa741b9aca6d9f1280c8af15\": container with ID starting with da4ecfcfedb18af3f9f5fd59d1f8e23d5103f2b2fa741b9aca6d9f1280c8af15 not found: ID does not exist" Sep 09 02:47:49.718230 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.718215 2577 scope.go:117] "RemoveContainer" containerID="d407e64082e62441f1717d784844db840e5d4162413cd0ebf70a1d6845f2ea81" Sep 09 02:47:49.718477 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:47:49.718460 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d407e64082e62441f1717d784844db840e5d4162413cd0ebf70a1d6845f2ea81\": container with ID starting with d407e64082e62441f1717d784844db840e5d4162413cd0ebf70a1d6845f2ea81 not found: ID does not exist" containerID="d407e64082e62441f1717d784844db840e5d4162413cd0ebf70a1d6845f2ea81" Sep 09 02:47:49.718515 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.718484 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d407e64082e62441f1717d784844db840e5d4162413cd0ebf70a1d6845f2ea81"} err="failed to get container status \"d407e64082e62441f1717d784844db840e5d4162413cd0ebf70a1d6845f2ea81\": rpc error: code = NotFound desc = could not find container \"d407e64082e62441f1717d784844db840e5d4162413cd0ebf70a1d6845f2ea81\": container with ID starting with d407e64082e62441f1717d784844db840e5d4162413cd0ebf70a1d6845f2ea81 not found: ID does not exist" Sep 09 02:47:49.728785 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.728763 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l"] Sep 09 02:47:49.731878 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:49.731856 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-1816a546-openshift-default-5fd94d5c56-s4r7l"] Sep 09 02:47:50.310730 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:50.310698 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" path="/var/lib/kubelet/pods/8c5d3c27-0cf6-4451-a437-f5d2f4fba41a/volumes" Sep 09 02:47:51.947718 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:51.947675 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:52.012441 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:52.012399 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:52.086256 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:52.086218 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:52.184924 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:52.184895 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:52.284678 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:52.284596 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:52.671787 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:52.671756 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:47:54.237711 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:47:54.237679 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:04.877151 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:04.877115 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:18.652756 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:18.652723 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:24.345992 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:24.345957 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:24.408248 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:24.408215 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:24.509993 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:24.509958 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:24.642199 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:24.642166 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:29.042959 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:29.042921 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:38.483377 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:38.483344 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:40.430203 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:40.430164 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:46.314015 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:46.313971 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:46.419675 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:46.419644 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:46.515491 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:46.515456 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:49.206682 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:49.206648 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695"] Sep 09 02:48:49.207200 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:49.206913 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" podUID="a981159c-f4ba-4d23-b4e7-54bec16872ee" containerName="maas-api" containerID="cri-o://7d6f411fe6723fa536f84947523a85a2e8869e345f6dbe13cf0ab4fad5209c4d" gracePeriod=30 Sep 09 02:48:49.238162 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:49.238135 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:49.315328 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:49.315291 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:49.382195 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:49.382168 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:49.445151 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:49.445123 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:49.469396 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:49.469332 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" Sep 09 02:48:49.542755 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:49.542725 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a981159c-f4ba-4d23-b4e7-54bec16872ee-openshift-service-ca\") pod \"a981159c-f4ba-4d23-b4e7-54bec16872ee\" (UID: \"a981159c-f4ba-4d23-b4e7-54bec16872ee\") " Sep 09 02:48:49.542947 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:49.542787 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a981159c-f4ba-4d23-b4e7-54bec16872ee-maas-api-tls\") pod \"a981159c-f4ba-4d23-b4e7-54bec16872ee\" (UID: \"a981159c-f4ba-4d23-b4e7-54bec16872ee\") " Sep 09 02:48:49.542947 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:49.542894 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86z5d\" (UniqueName: \"kubernetes.io/projected/a981159c-f4ba-4d23-b4e7-54bec16872ee-kube-api-access-86z5d\") pod \"a981159c-f4ba-4d23-b4e7-54bec16872ee\" (UID: \"a981159c-f4ba-4d23-b4e7-54bec16872ee\") " Sep 09 02:48:49.542947 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:49.542941 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a981159c-f4ba-4d23-b4e7-54bec16872ee-metrics-certs\") pod \"a981159c-f4ba-4d23-b4e7-54bec16872ee\" (UID: \"a981159c-f4ba-4d23-b4e7-54bec16872ee\") " Sep 09 02:48:49.543172 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:49.543146 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a981159c-f4ba-4d23-b4e7-54bec16872ee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a981159c-f4ba-4d23-b4e7-54bec16872ee" (UID: "a981159c-f4ba-4d23-b4e7-54bec16872ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:48:49.543293 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:49.543271 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a981159c-f4ba-4d23-b4e7-54bec16872ee-openshift-service-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:48:49.545004 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:49.544969 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a981159c-f4ba-4d23-b4e7-54bec16872ee-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a981159c-f4ba-4d23-b4e7-54bec16872ee" (UID: "a981159c-f4ba-4d23-b4e7-54bec16872ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:48:49.545004 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:49.544986 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a981159c-f4ba-4d23-b4e7-54bec16872ee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a981159c-f4ba-4d23-b4e7-54bec16872ee" (UID: "a981159c-f4ba-4d23-b4e7-54bec16872ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:48:49.545122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:49.545031 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a981159c-f4ba-4d23-b4e7-54bec16872ee-kube-api-access-86z5d" (OuterVolumeSpecName: "kube-api-access-86z5d") pod "a981159c-f4ba-4d23-b4e7-54bec16872ee" (UID: "a981159c-f4ba-4d23-b4e7-54bec16872ee"). InnerVolumeSpecName "kube-api-access-86z5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:48:49.644252 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:49.644218 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a981159c-f4ba-4d23-b4e7-54bec16872ee-maas-api-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:48:49.644252 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:49.644248 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86z5d\" (UniqueName: \"kubernetes.io/projected/a981159c-f4ba-4d23-b4e7-54bec16872ee-kube-api-access-86z5d\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:48:49.644252 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:49.644258 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a981159c-f4ba-4d23-b4e7-54bec16872ee-metrics-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:48:49.816944 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:49.816860 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:50.017707 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:50.017673 2577 generic.go:356] "Generic (PLEG): container finished" podID="a981159c-f4ba-4d23-b4e7-54bec16872ee" containerID="7d6f411fe6723fa536f84947523a85a2e8869e345f6dbe13cf0ab4fad5209c4d" exitCode=0 Sep 09 02:48:50.017902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:50.017719 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" Sep 09 02:48:50.017902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:50.017760 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" event={"ID":"a981159c-f4ba-4d23-b4e7-54bec16872ee","Type":"ContainerDied","Data":"7d6f411fe6723fa536f84947523a85a2e8869e345f6dbe13cf0ab4fad5209c4d"} Sep 09 02:48:50.017902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:50.017831 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695" event={"ID":"a981159c-f4ba-4d23-b4e7-54bec16872ee","Type":"ContainerDied","Data":"ddb6aa0261f54ee87569f1781acb3fccffd9988990091d1ad2f2c98f62bcc63a"} Sep 09 02:48:50.017902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:50.017862 2577 scope.go:117] "RemoveContainer" containerID="7d6f411fe6723fa536f84947523a85a2e8869e345f6dbe13cf0ab4fad5209c4d" Sep 09 02:48:50.027190 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:50.027173 2577 scope.go:117] "RemoveContainer" containerID="7d6f411fe6723fa536f84947523a85a2e8869e345f6dbe13cf0ab4fad5209c4d" Sep 09 02:48:50.027438 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:48:50.027418 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d6f411fe6723fa536f84947523a85a2e8869e345f6dbe13cf0ab4fad5209c4d\": container with ID starting with 7d6f411fe6723fa536f84947523a85a2e8869e345f6dbe13cf0ab4fad5209c4d not found: ID does not exist" containerID="7d6f411fe6723fa536f84947523a85a2e8869e345f6dbe13cf0ab4fad5209c4d" Sep 09 02:48:50.027503 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:50.027450 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d6f411fe6723fa536f84947523a85a2e8869e345f6dbe13cf0ab4fad5209c4d"} err="failed to get container status \"7d6f411fe6723fa536f84947523a85a2e8869e345f6dbe13cf0ab4fad5209c4d\": rpc error: code = NotFound desc = could not find container \"7d6f411fe6723fa536f84947523a85a2e8869e345f6dbe13cf0ab4fad5209c4d\": container with ID starting with 7d6f411fe6723fa536f84947523a85a2e8869e345f6dbe13cf0ab4fad5209c4d not found: ID does not exist" Sep 09 02:48:50.040330 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:50.040305 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695"] Sep 09 02:48:50.044424 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:50.044405 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b52336-77df5c888-87695"] Sep 09 02:48:50.311291 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:50.311253 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a981159c-f4ba-4d23-b4e7-54bec16872ee" path="/var/lib/kubelet/pods/a981159c-f4ba-4d23-b4e7-54bec16872ee/volumes" Sep 09 02:48:55.647494 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:55.647462 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:55.748646 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:55.748615 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:55.920686 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:55.920611 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:56.383648 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.383614 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:56.684893 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.680783 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s"] Sep 09 02:48:56.684893 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.681830 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a981159c-f4ba-4d23-b4e7-54bec16872ee" containerName="maas-api" Sep 09 02:48:56.684893 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.681853 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a981159c-f4ba-4d23-b4e7-54bec16872ee" containerName="maas-api" Sep 09 02:48:56.684893 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.681887 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" containerName="istio-proxy" Sep 09 02:48:56.684893 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.681896 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" containerName="istio-proxy" Sep 09 02:48:56.684893 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.681912 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f7b93aa-f8e9-4dc5-9003-cfe75193e969" containerName="maas-api" Sep 09 02:48:56.684893 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.681921 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f7b93aa-f8e9-4dc5-9003-cfe75193e969" containerName="maas-api" Sep 09 02:48:56.684893 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.681948 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b5deaea-bded-4181-99aa-69d4a867daaa" containerName="istio-proxy" Sep 09 02:48:56.684893 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.681956 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5deaea-bded-4181-99aa-69d4a867daaa" containerName="istio-proxy" Sep 09 02:48:56.684893 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.682012 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" containerName="istio-proxy" Sep 09 02:48:56.684893 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.682021 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" containerName="istio-proxy" Sep 09 02:48:56.684893 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.682209 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f7b93aa-f8e9-4dc5-9003-cfe75193e969" containerName="maas-api" Sep 09 02:48:56.684893 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.682220 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b5deaea-bded-4181-99aa-69d4a867daaa" containerName="istio-proxy" Sep 09 02:48:56.684893 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.682240 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b5deaea-bded-4181-99aa-69d4a867daaa" containerName="istio-proxy" Sep 09 02:48:56.684893 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.682249 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" containerName="istio-proxy" Sep 09 02:48:56.684893 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.682267 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c5d3c27-0cf6-4451-a437-f5d2f4fba41a" containerName="istio-proxy" Sep 09 02:48:56.684893 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.682283 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a981159c-f4ba-4d23-b4e7-54bec16872ee" containerName="maas-api" Sep 09 02:48:56.688220 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.688191 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.691363 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.691340 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-ac6878-openshift-default-dockercfg-dgj6t\"" Sep 09 02:48:56.693222 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.693200 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s"] Sep 09 02:48:56.701948 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.701923 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:56.817101 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.817068 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-credential-socket\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.817101 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.817100 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-token\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.817346 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.817139 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e90514e8-9cdb-40c2-9deb-df46033e98a4-istiod-ca-cert\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.817346 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.817171 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-data\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.817346 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.817193 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-workload-socket\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.817346 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.817220 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lt7h\" (UniqueName: \"kubernetes.io/projected/e90514e8-9cdb-40c2-9deb-df46033e98a4-kube-api-access-5lt7h\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.817346 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.817255 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-envoy\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.817346 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.817283 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-podinfo\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.817578 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.817364 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-workload-certs\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.918416 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.918371 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-workload-certs\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.918621 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.918472 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-credential-socket\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.918621 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.918500 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-token\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.918621 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.918541 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e90514e8-9cdb-40c2-9deb-df46033e98a4-istiod-ca-cert\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.918621 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.918568 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-data\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.918621 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.918610 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-workload-socket\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.918941 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.918647 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lt7h\" (UniqueName: \"kubernetes.io/projected/e90514e8-9cdb-40c2-9deb-df46033e98a4-kube-api-access-5lt7h\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.918941 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.918701 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-envoy\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.918941 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.918751 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-podinfo\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.918941 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.918862 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-workload-certs\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.919155 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.918944 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-data\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.919155 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.919089 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-workload-socket\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.919257 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.919215 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-credential-socket\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.919257 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.919246 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e90514e8-9cdb-40c2-9deb-df46033e98a4-istiod-ca-cert\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.921086 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.921063 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-envoy\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.921329 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.921308 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-podinfo\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.926748 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.926727 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lt7h\" (UniqueName: \"kubernetes.io/projected/e90514e8-9cdb-40c2-9deb-df46033e98a4-kube-api-access-5lt7h\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.926855 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.926770 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-token\") pod \"e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:56.992327 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:56.992239 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:57.000036 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:57.000005 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:57.091692 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:57.091662 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:57.284603 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:57.284572 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s"] Sep 09 02:48:57.286105 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:48:57.286079 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90514e8_9cdb_40c2_9deb_df46033e98a4.slice/crio-01b930dd60b006b9d2b8593ea3aafabc1fc82987b62fb63dade56a243974bd85 WatchSource:0}: Error finding container 01b930dd60b006b9d2b8593ea3aafabc1fc82987b62fb63dade56a243974bd85: Status 404 returned error can't find the container with id 01b930dd60b006b9d2b8593ea3aafabc1fc82987b62fb63dade56a243974bd85 Sep 09 02:48:57.288213 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:57.288181 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:48:57.288328 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:57.288264 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:48:57.288328 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:57.288310 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:48:57.841630 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:57.841596 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:57.972076 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:57.972047 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:58.064377 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:58.064341 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" event={"ID":"e90514e8-9cdb-40c2-9deb-df46033e98a4","Type":"ContainerStarted","Data":"9817ff57c370ca853bb4790fe47fc200792122c7f715cb64c3db60f714078037"} Sep 09 02:48:58.064579 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:58.064399 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" event={"ID":"e90514e8-9cdb-40c2-9deb-df46033e98a4","Type":"ContainerStarted","Data":"01b930dd60b006b9d2b8593ea3aafabc1fc82987b62fb63dade56a243974bd85"} Sep 09 02:48:58.086172 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:58.086118 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" podStartSLOduration=2.086097911 podStartE2EDuration="2.086097911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:48:58.085467147 +0000 UTC m=+1126.215709257" watchObservedRunningTime="2026-09-09 02:48:58.086097911 +0000 UTC m=+1126.216340007" Sep 09 02:48:58.360957 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:58.360924 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:58.469600 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:58.469564 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:58.637873 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:58.637841 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:48:59.001313 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:59.001223 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:59.006223 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:59.006195 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:59.069650 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:59.069614 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:48:59.070580 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:48:59.070555 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:49:03.935650 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:03.935616 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:04.046663 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:04.046634 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:04.139417 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:04.139383 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:04.237343 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:04.237256 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:04.972834 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:04.972788 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:05.074688 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:05.074643 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:05.176747 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:05.176717 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:06.720399 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:06.720363 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:09.584888 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:09.584847 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:12.434224 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:12.434186 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:12.534305 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:12.534277 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:13.118305 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:13.118269 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:19.680162 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:19.680126 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:19.759732 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:19.759703 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:19.924704 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:19.924670 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:20.192011 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.191977 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v"] Sep 09 02:49:20.192688 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.192673 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b5deaea-bded-4181-99aa-69d4a867daaa" containerName="istio-proxy" Sep 09 02:49:20.192742 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.192692 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5deaea-bded-4181-99aa-69d4a867daaa" containerName="istio-proxy" Sep 09 02:49:20.197745 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.197724 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.200370 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.200347 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-81f1254e-openshift-default-dockercfg-xbm5v\"" Sep 09 02:49:20.207711 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.207688 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v"] Sep 09 02:49:20.222345 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.222317 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:20.243495 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.243437 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-token\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.243677 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.243541 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-workload-certs\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.243677 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.243599 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz4wh\" (UniqueName: \"kubernetes.io/projected/aea90141-6b57-4ddb-bba4-046d278fe07e-kube-api-access-dz4wh\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.243677 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.243657 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-podinfo\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.243874 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.243704 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-envoy\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.243874 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.243733 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-credential-socket\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.243874 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.243780 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-workload-socket\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.243874 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.243851 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea90141-6b57-4ddb-bba4-046d278fe07e-istiod-ca-cert\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.244090 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.243897 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-data\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.318787 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.318749 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:20.344755 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.344725 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-credential-socket\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.344929 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.344765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-workload-socket\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.344929 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.344784 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea90141-6b57-4ddb-bba4-046d278fe07e-istiod-ca-cert\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.344929 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.344877 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-data\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.345056 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.345002 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-token\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.345056 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.345034 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-workload-certs\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.345161 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.345084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz4wh\" (UniqueName: \"kubernetes.io/projected/aea90141-6b57-4ddb-bba4-046d278fe07e-kube-api-access-dz4wh\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.345161 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.345117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-podinfo\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.345265 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.345162 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-envoy\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.345265 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.345194 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-workload-socket\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.345265 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.345221 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-credential-socket\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.345421 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.345285 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-data\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.345421 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.345370 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-workload-certs\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.345534 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.345510 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea90141-6b57-4ddb-bba4-046d278fe07e-istiod-ca-cert\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.347437 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.347408 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-envoy\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.347522 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.347505 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-podinfo\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.356434 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.356409 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-token\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.357636 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.357612 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz4wh\" (UniqueName: \"kubernetes.io/projected/aea90141-6b57-4ddb-bba4-046d278fe07e-kube-api-access-dz4wh\") pod \"e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.509954 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.509869 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:20.791514 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.791485 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v"] Sep 09 02:49:20.792923 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:49:20.792893 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea90141_6b57_4ddb_bba4_046d278fe07e.slice/crio-4c665256eb83ddefeda3af25938705c2184c0cf55006a4fa9c6224bb3b084dfd WatchSource:0}: Error finding container 4c665256eb83ddefeda3af25938705c2184c0cf55006a4fa9c6224bb3b084dfd: Status 404 returned error can't find the container with id 4c665256eb83ddefeda3af25938705c2184c0cf55006a4fa9c6224bb3b084dfd Sep 09 02:49:20.795095 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.795065 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:49:20.795208 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.795141 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:49:20.795208 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:20.795187 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:49:21.183599 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:21.183563 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" event={"ID":"aea90141-6b57-4ddb-bba4-046d278fe07e","Type":"ContainerStarted","Data":"04ded24bdf1c99970e035c8f96d432ffbceede51c96666dbb641d2fd74e40319"} Sep 09 02:49:21.183761 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:21.183621 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" event={"ID":"aea90141-6b57-4ddb-bba4-046d278fe07e","Type":"ContainerStarted","Data":"4c665256eb83ddefeda3af25938705c2184c0cf55006a4fa9c6224bb3b084dfd"} Sep 09 02:49:21.205848 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:21.205782 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" podStartSLOduration=1.205765271 podStartE2EDuration="1.205765271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:49:21.203777182 +0000 UTC m=+1149.334019278" watchObservedRunningTime="2026-09-09 02:49:21.205765271 +0000 UTC m=+1149.336007366" Sep 09 02:49:21.510909 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:21.510802 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:21.515826 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:21.515787 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:22.189206 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:22.189178 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:22.190127 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:22.190106 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:23.136896 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:23.136856 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:23.411455 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:23.411370 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:24.423970 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:24.423936 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447"] Sep 09 02:49:24.428956 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:24.428305 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" Sep 09 02:49:24.434298 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:24.434271 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-81f1254e-serving-cert\"" Sep 09 02:49:24.441092 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:24.441054 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447"] Sep 09 02:49:24.461772 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:24.461741 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:24.489042 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:24.489007 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7pdd\" (UniqueName: \"kubernetes.io/projected/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-kube-api-access-c7pdd\") pod \"maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447\" (UID: \"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" Sep 09 02:49:24.489186 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:24.489045 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-openshift-service-ca\") pod \"maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447\" (UID: \"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" Sep 09 02:49:24.489186 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:24.489092 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-maas-api-tls\") pod \"maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447\" (UID: \"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" Sep 09 02:49:24.489186 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:24.489145 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-metrics-certs\") pod \"maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447\" (UID: \"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" Sep 09 02:49:24.562527 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:24.562497 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:24.589926 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:24.589895 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7pdd\" (UniqueName: \"kubernetes.io/projected/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-kube-api-access-c7pdd\") pod \"maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447\" (UID: \"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" Sep 09 02:49:24.590092 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:24.589933 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-openshift-service-ca\") pod \"maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447\" (UID: \"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" Sep 09 02:49:24.590092 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:24.589982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-maas-api-tls\") pod \"maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447\" (UID: \"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" Sep 09 02:49:24.590207 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:24.590138 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-metrics-certs\") pod \"maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447\" (UID: \"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" Sep 09 02:49:24.590718 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:24.590698 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-openshift-service-ca\") pod \"maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447\" (UID: \"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" Sep 09 02:49:24.592402 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:24.592382 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-maas-api-tls\") pod \"maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447\" (UID: \"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" Sep 09 02:49:24.592511 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:24.592492 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-metrics-certs\") pod \"maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447\" (UID: \"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" Sep 09 02:49:24.597992 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:24.597974 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7pdd\" (UniqueName: \"kubernetes.io/projected/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-kube-api-access-c7pdd\") pod \"maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447\" (UID: \"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" Sep 09 02:49:24.663320 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:24.663287 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:24.746689 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:24.746615 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" Sep 09 02:49:24.944620 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:24.944578 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:25.023304 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:25.023279 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447"] Sep 09 02:49:25.025257 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:49:25.025227 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f1c5f5_124d_4519_9bdd_6fd7adefa0d7.slice/crio-1551e02b4bb15c61d182de62884cb356a008fce59b27209f408d62eb4c59eab4 WatchSource:0}: Error finding container 1551e02b4bb15c61d182de62884cb356a008fce59b27209f408d62eb4c59eab4: Status 404 returned error can't find the container with id 1551e02b4bb15c61d182de62884cb356a008fce59b27209f408d62eb4c59eab4 Sep 09 02:49:25.025882 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:25.025856 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:25.205703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:25.205668 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" event={"ID":"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7","Type":"ContainerStarted","Data":"1551e02b4bb15c61d182de62884cb356a008fce59b27209f408d62eb4c59eab4"} Sep 09 02:49:26.212882 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:26.212845 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" event={"ID":"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7","Type":"ContainerStarted","Data":"650d9038e64ad5d7b85084798c3611f64517e15f4c050a2776a0e70e314be33d"} Sep 09 02:49:26.213326 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:26.212922 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" Sep 09 02:49:26.232670 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:26.232616 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" podStartSLOduration=1.819604472 podStartE2EDuration="2.232598096s" totalImagesPullingTime="412.993624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:49:26.231853258 +0000 UTC m=+1154.362095355" watchObservedRunningTime="2026-09-09 02:49:26.232598096 +0000 UTC m=+1154.362840191" Sep 09 02:49:32.224323 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:32.224290 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" Sep 09 02:49:33.550190 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:33.550158 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:34.845433 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:34.845396 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:34.943130 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:34.943053 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s"] Sep 09 02:49:34.943354 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:34.943298 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" podUID="e90514e8-9cdb-40c2-9deb-df46033e98a4" containerName="istio-proxy" containerID="cri-o://9817ff57c370ca853bb4790fe47fc200792122c7f715cb64c3db60f714078037" gracePeriod=30 Sep 09 02:49:34.966922 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:34.966883 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:35.213213 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:35.213175 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:35.314676 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:35.314645 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:36.744925 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:36.744892 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:37.113039 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:37.112958 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:40.198596 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.198561 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:49:40.299423 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.299333 2577 generic.go:356] "Generic (PLEG): container finished" podID="e90514e8-9cdb-40c2-9deb-df46033e98a4" containerID="9817ff57c370ca853bb4790fe47fc200792122c7f715cb64c3db60f714078037" exitCode=0 Sep 09 02:49:40.299423 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.299383 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" Sep 09 02:49:40.299423 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.299398 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" event={"ID":"e90514e8-9cdb-40c2-9deb-df46033e98a4","Type":"ContainerDied","Data":"9817ff57c370ca853bb4790fe47fc200792122c7f715cb64c3db60f714078037"} Sep 09 02:49:40.299696 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.299458 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s" event={"ID":"e90514e8-9cdb-40c2-9deb-df46033e98a4","Type":"ContainerDied","Data":"01b930dd60b006b9d2b8593ea3aafabc1fc82987b62fb63dade56a243974bd85"} Sep 09 02:49:40.299696 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.299490 2577 scope.go:117] "RemoveContainer" containerID="9817ff57c370ca853bb4790fe47fc200792122c7f715cb64c3db60f714078037" Sep 09 02:49:40.310718 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.310693 2577 scope.go:117] "RemoveContainer" containerID="9817ff57c370ca853bb4790fe47fc200792122c7f715cb64c3db60f714078037" Sep 09 02:49:40.311030 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:49:40.311006 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9817ff57c370ca853bb4790fe47fc200792122c7f715cb64c3db60f714078037\": container with ID starting with 9817ff57c370ca853bb4790fe47fc200792122c7f715cb64c3db60f714078037 not found: ID does not exist" containerID="9817ff57c370ca853bb4790fe47fc200792122c7f715cb64c3db60f714078037" Sep 09 02:49:40.311148 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.311042 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9817ff57c370ca853bb4790fe47fc200792122c7f715cb64c3db60f714078037"} err="failed to get container status \"9817ff57c370ca853bb4790fe47fc200792122c7f715cb64c3db60f714078037\": rpc error: code = NotFound desc = could not find container \"9817ff57c370ca853bb4790fe47fc200792122c7f715cb64c3db60f714078037\": container with ID starting with 9817ff57c370ca853bb4790fe47fc200792122c7f715cb64c3db60f714078037 not found: ID does not exist" Sep 09 02:49:40.329718 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.329692 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:40.352929 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.352888 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-workload-socket\") pod \"e90514e8-9cdb-40c2-9deb-df46033e98a4\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " Sep 09 02:49:40.352929 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.352927 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-workload-certs\") pod \"e90514e8-9cdb-40c2-9deb-df46033e98a4\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " Sep 09 02:49:40.353181 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.352959 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e90514e8-9cdb-40c2-9deb-df46033e98a4-istiod-ca-cert\") pod \"e90514e8-9cdb-40c2-9deb-df46033e98a4\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " Sep 09 02:49:40.353181 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.352984 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-data\") pod \"e90514e8-9cdb-40c2-9deb-df46033e98a4\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " Sep 09 02:49:40.353181 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.353027 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-credential-socket\") pod \"e90514e8-9cdb-40c2-9deb-df46033e98a4\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " Sep 09 02:49:40.353181 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.353066 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-token\") pod \"e90514e8-9cdb-40c2-9deb-df46033e98a4\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " Sep 09 02:49:40.353181 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.353124 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-podinfo\") pod \"e90514e8-9cdb-40c2-9deb-df46033e98a4\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " Sep 09 02:49:40.353399 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.353213 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e90514e8-9cdb-40c2-9deb-df46033e98a4" (UID: "e90514e8-9cdb-40c2-9deb-df46033e98a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:49:40.353399 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.353256 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e90514e8-9cdb-40c2-9deb-df46033e98a4" (UID: "e90514e8-9cdb-40c2-9deb-df46033e98a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:49:40.353399 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.353362 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-envoy\") pod \"e90514e8-9cdb-40c2-9deb-df46033e98a4\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " Sep 09 02:49:40.353399 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.353370 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-data" (OuterVolumeSpecName: "istio-data") pod "e90514e8-9cdb-40c2-9deb-df46033e98a4" (UID: "e90514e8-9cdb-40c2-9deb-df46033e98a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:49:40.353616 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.353401 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e90514e8-9cdb-40c2-9deb-df46033e98a4" (UID: "e90514e8-9cdb-40c2-9deb-df46033e98a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:49:40.353616 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.353421 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lt7h\" (UniqueName: \"kubernetes.io/projected/e90514e8-9cdb-40c2-9deb-df46033e98a4-kube-api-access-5lt7h\") pod \"e90514e8-9cdb-40c2-9deb-df46033e98a4\" (UID: \"e90514e8-9cdb-40c2-9deb-df46033e98a4\") " Sep 09 02:49:40.353616 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.353490 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90514e8-9cdb-40c2-9deb-df46033e98a4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e90514e8-9cdb-40c2-9deb-df46033e98a4" (UID: "e90514e8-9cdb-40c2-9deb-df46033e98a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:49:40.353945 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.353873 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-workload-socket\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:49:40.353945 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.353899 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-workload-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:49:40.353945 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.353915 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e90514e8-9cdb-40c2-9deb-df46033e98a4-istiod-ca-cert\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:49:40.353945 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.353930 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-data\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:49:40.353945 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.353943 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-credential-socket\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:49:40.355674 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.355647 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e90514e8-9cdb-40c2-9deb-df46033e98a4" (UID: "e90514e8-9cdb-40c2-9deb-df46033e98a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:49:40.355797 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.355769 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-token" (OuterVolumeSpecName: "istio-token") pod "e90514e8-9cdb-40c2-9deb-df46033e98a4" (UID: "e90514e8-9cdb-40c2-9deb-df46033e98a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:49:40.355880 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.355796 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e90514e8-9cdb-40c2-9deb-df46033e98a4" (UID: "e90514e8-9cdb-40c2-9deb-df46033e98a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 02:49:40.356081 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.356060 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90514e8-9cdb-40c2-9deb-df46033e98a4-kube-api-access-5lt7h" (OuterVolumeSpecName: "kube-api-access-5lt7h") pod "e90514e8-9cdb-40c2-9deb-df46033e98a4" (UID: "e90514e8-9cdb-40c2-9deb-df46033e98a4"). InnerVolumeSpecName "kube-api-access-5lt7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:49:40.455393 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.455357 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-token\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:49:40.455393 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.455394 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-podinfo\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:49:40.455600 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.455407 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e90514e8-9cdb-40c2-9deb-df46033e98a4-istio-envoy\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:49:40.455600 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.455420 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lt7h\" (UniqueName: \"kubernetes.io/projected/e90514e8-9cdb-40c2-9deb-df46033e98a4-kube-api-access-5lt7h\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:49:40.623406 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.623371 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s"] Sep 09 02:49:40.628084 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:40.628051 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-ac6878-openshift-default-67b5bd9874-fqn4s"] Sep 09 02:49:42.311581 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:42.311550 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e90514e8-9cdb-40c2-9deb-df46033e98a4" path="/var/lib/kubelet/pods/e90514e8-9cdb-40c2-9deb-df46033e98a4/volumes" Sep 09 02:49:43.328413 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:43.328380 2577 generic.go:356] "Generic (PLEG): container finished" podID="aea90141-6b57-4ddb-bba4-046d278fe07e" containerID="04ded24bdf1c99970e035c8f96d432ffbceede51c96666dbb641d2fd74e40319" exitCode=0 Sep 09 02:49:43.328800 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:43.328461 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" event={"ID":"aea90141-6b57-4ddb-bba4-046d278fe07e","Type":"ContainerDied","Data":"04ded24bdf1c99970e035c8f96d432ffbceede51c96666dbb641d2fd74e40319"} Sep 09 02:49:43.329041 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:43.329024 2577 scope.go:117] "RemoveContainer" containerID="04ded24bdf1c99970e035c8f96d432ffbceede51c96666dbb641d2fd74e40319" Sep 09 02:49:43.330211 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:43.330185 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:49:43.330283 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:43.330269 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:49:43.330327 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:43.330309 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:49:44.340982 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:44.339507 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" event={"ID":"aea90141-6b57-4ddb-bba4-046d278fe07e","Type":"ContainerStarted","Data":"e6aa2e368b0fc2b5ce56b69e47811f1a1df546ed047c5f7518acceb64fae963b"} Sep 09 02:49:44.510609 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:44.510569 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:44.510914 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:44.510888 2577 patch_prober.go:28] interesting pod/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 09 02:49:44.510968 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:44.510948 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" podUID="aea90141-6b57-4ddb-bba4-046d278fe07e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 09 02:49:45.406834 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:45.406782 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:45.510901 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:45.510863 2577 patch_prober.go:28] interesting pod/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 09 02:49:45.511080 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:45.510923 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" podUID="aea90141-6b57-4ddb-bba4-046d278fe07e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 09 02:49:46.510979 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:46.510942 2577 patch_prober.go:28] interesting pod/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 09 02:49:46.511414 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:46.511014 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" podUID="aea90141-6b57-4ddb-bba4-046d278fe07e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 09 02:49:47.238734 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.238701 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2"] Sep 09 02:49:47.239034 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.238970 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" podUID="fb7aebac-c330-497e-a0bd-ed44095a0486" containerName="maas-api" containerID="cri-o://815d6f5f03f501ae20a5675b25764ba4d9d31e78469c91fe9d210d516d2c587e" gracePeriod=30 Sep 09 02:49:47.288296 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.288158 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:47.382105 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.382057 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:47.385628 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.385502 2577 generic.go:356] "Generic (PLEG): container finished" podID="fb7aebac-c330-497e-a0bd-ed44095a0486" containerID="815d6f5f03f501ae20a5675b25764ba4d9d31e78469c91fe9d210d516d2c587e" exitCode=0 Sep 09 02:49:47.385771 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.385624 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" event={"ID":"fb7aebac-c330-497e-a0bd-ed44095a0486","Type":"ContainerDied","Data":"815d6f5f03f501ae20a5675b25764ba4d9d31e78469c91fe9d210d516d2c587e"} Sep 09 02:49:47.446769 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.446662 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:47.511272 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.511186 2577 patch_prober.go:28] interesting pod/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 09 02:49:47.511272 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.511241 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" podUID="aea90141-6b57-4ddb-bba4-046d278fe07e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 09 02:49:47.534498 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.534466 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:47.543961 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.543936 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" Sep 09 02:49:47.596324 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.596294 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:47.624699 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.624665 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7aebac-c330-497e-a0bd-ed44095a0486-metrics-certs\") pod \"fb7aebac-c330-497e-a0bd-ed44095a0486\" (UID: \"fb7aebac-c330-497e-a0bd-ed44095a0486\") " Sep 09 02:49:47.624890 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.624723 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmfkw\" (UniqueName: \"kubernetes.io/projected/fb7aebac-c330-497e-a0bd-ed44095a0486-kube-api-access-zmfkw\") pod \"fb7aebac-c330-497e-a0bd-ed44095a0486\" (UID: \"fb7aebac-c330-497e-a0bd-ed44095a0486\") " Sep 09 02:49:47.624890 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.624782 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb7aebac-c330-497e-a0bd-ed44095a0486-maas-api-tls\") pod \"fb7aebac-c330-497e-a0bd-ed44095a0486\" (UID: \"fb7aebac-c330-497e-a0bd-ed44095a0486\") " Sep 09 02:49:47.624890 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.624879 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb7aebac-c330-497e-a0bd-ed44095a0486-openshift-service-ca\") pod \"fb7aebac-c330-497e-a0bd-ed44095a0486\" (UID: \"fb7aebac-c330-497e-a0bd-ed44095a0486\") " Sep 09 02:49:47.625481 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.625432 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb7aebac-c330-497e-a0bd-ed44095a0486-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fb7aebac-c330-497e-a0bd-ed44095a0486" (UID: "fb7aebac-c330-497e-a0bd-ed44095a0486"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:49:47.627379 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.627348 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb7aebac-c330-497e-a0bd-ed44095a0486-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fb7aebac-c330-497e-a0bd-ed44095a0486" (UID: "fb7aebac-c330-497e-a0bd-ed44095a0486"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:49:47.627491 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.627385 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7aebac-c330-497e-a0bd-ed44095a0486-kube-api-access-zmfkw" (OuterVolumeSpecName: "kube-api-access-zmfkw") pod "fb7aebac-c330-497e-a0bd-ed44095a0486" (UID: "fb7aebac-c330-497e-a0bd-ed44095a0486"). InnerVolumeSpecName "kube-api-access-zmfkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:49:47.627491 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.627404 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb7aebac-c330-497e-a0bd-ed44095a0486-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fb7aebac-c330-497e-a0bd-ed44095a0486" (UID: "fb7aebac-c330-497e-a0bd-ed44095a0486"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:49:47.725640 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.725603 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7aebac-c330-497e-a0bd-ed44095a0486-metrics-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:49:47.725640 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.725633 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmfkw\" (UniqueName: \"kubernetes.io/projected/fb7aebac-c330-497e-a0bd-ed44095a0486-kube-api-access-zmfkw\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:49:47.725640 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.725645 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb7aebac-c330-497e-a0bd-ed44095a0486-maas-api-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:49:47.725899 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.725654 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb7aebac-c330-497e-a0bd-ed44095a0486-openshift-service-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:49:47.893490 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:47.893461 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:48.102513 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:48.102480 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:48.392163 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:48.392130 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" Sep 09 02:49:48.392359 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:48.392175 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2" event={"ID":"fb7aebac-c330-497e-a0bd-ed44095a0486","Type":"ContainerDied","Data":"ca7057216c2f3001654150356a73bd286feffe5d03c81a343496342ad05654f9"} Sep 09 02:49:48.392359 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:48.392251 2577 scope.go:117] "RemoveContainer" containerID="815d6f5f03f501ae20a5675b25764ba4d9d31e78469c91fe9d210d516d2c587e" Sep 09 02:49:48.409098 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:48.409074 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2"] Sep 09 02:49:48.412437 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:48.412418 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b18109-7f6884b4dc-mxgl2"] Sep 09 02:49:48.510484 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:48.510451 2577 patch_prober.go:28] interesting pod/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 09 02:49:48.510658 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:48.510534 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" podUID="aea90141-6b57-4ddb-bba4-046d278fe07e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 09 02:49:48.772091 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:48.772058 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:48.887884 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:48.887853 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:49.225077 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:49.225045 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:49.510512 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:49.510430 2577 patch_prober.go:28] interesting pod/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 09 02:49:49.510512 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:49.510500 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" podUID="aea90141-6b57-4ddb-bba4-046d278fe07e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 09 02:49:50.310538 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:50.310500 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb7aebac-c330-497e-a0bd-ed44095a0486" path="/var/lib/kubelet/pods/fb7aebac-c330-497e-a0bd-ed44095a0486/volumes" Sep 09 02:49:50.516975 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:50.516943 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:50.517290 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:50.517268 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:50.518126 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:50.518094 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:49:53.986524 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:53.986490 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:54.088825 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:54.088782 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:54.218356 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:54.218326 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8"] Sep 09 02:49:54.218592 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:54.218559 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" podUID="aee01b31-2432-4494-a93c-20cd8e73fefa" containerName="istio-proxy" containerID="cri-o://4cb7cc8052fc295ade6473ab98b1575716f29045e34c41435f8a766cf53b7ec1" gracePeriod=30 Sep 09 02:49:54.245296 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:54.245213 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:54.705343 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:54.705308 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:55.483867 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:55.483811 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:55.584842 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:55.584779 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:56.067215 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:56.067183 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:49:59.453390 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.453361 2577 generic.go:356] "Generic (PLEG): container finished" podID="aee01b31-2432-4494-a93c-20cd8e73fefa" containerID="4cb7cc8052fc295ade6473ab98b1575716f29045e34c41435f8a766cf53b7ec1" exitCode=0 Sep 09 02:49:59.453734 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.453430 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" event={"ID":"aee01b31-2432-4494-a93c-20cd8e73fefa","Type":"ContainerDied","Data":"4cb7cc8052fc295ade6473ab98b1575716f29045e34c41435f8a766cf53b7ec1"} Sep 09 02:49:59.453734 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.453496 2577 scope.go:117] "RemoveContainer" containerID="3809d0a7eac02e3f1c90d1ebe8a102e89da24878ebf2656746657e6df150c042" Sep 09 02:49:59.471656 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:49:59.471634 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3809d0a7eac02e3f1c90d1ebe8a102e89da24878ebf2656746657e6df150c042\": container with ID starting with 3809d0a7eac02e3f1c90d1ebe8a102e89da24878ebf2656746657e6df150c042 not found: ID does not exist" containerID="3809d0a7eac02e3f1c90d1ebe8a102e89da24878ebf2656746657e6df150c042" Sep 09 02:49:59.475223 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.475202 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:49:59.542261 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.542176 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-podinfo\") pod \"aee01b31-2432-4494-a93c-20cd8e73fefa\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " Sep 09 02:49:59.542261 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.542222 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-workload-certs\") pod \"aee01b31-2432-4494-a93c-20cd8e73fefa\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " Sep 09 02:49:59.542488 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.542289 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee01b31-2432-4494-a93c-20cd8e73fefa-istiod-ca-cert\") pod \"aee01b31-2432-4494-a93c-20cd8e73fefa\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " Sep 09 02:49:59.542488 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.542314 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-envoy\") pod \"aee01b31-2432-4494-a93c-20cd8e73fefa\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " Sep 09 02:49:59.542488 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.542369 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kd6sz\" (UniqueName: \"kubernetes.io/projected/aee01b31-2432-4494-a93c-20cd8e73fefa-kube-api-access-kd6sz\") pod \"aee01b31-2432-4494-a93c-20cd8e73fefa\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " Sep 09 02:49:59.542488 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.542405 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-credential-socket\") pod \"aee01b31-2432-4494-a93c-20cd8e73fefa\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " Sep 09 02:49:59.542488 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.542452 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-data\") pod \"aee01b31-2432-4494-a93c-20cd8e73fefa\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " Sep 09 02:49:59.542488 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.542476 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-workload-socket\") pod \"aee01b31-2432-4494-a93c-20cd8e73fefa\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " Sep 09 02:49:59.542795 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.542505 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-token\") pod \"aee01b31-2432-4494-a93c-20cd8e73fefa\" (UID: \"aee01b31-2432-4494-a93c-20cd8e73fefa\") " Sep 09 02:49:59.542795 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.542626 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aee01b31-2432-4494-a93c-20cd8e73fefa" (UID: "aee01b31-2432-4494-a93c-20cd8e73fefa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:49:59.542795 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.542703 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee01b31-2432-4494-a93c-20cd8e73fefa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aee01b31-2432-4494-a93c-20cd8e73fefa" (UID: "aee01b31-2432-4494-a93c-20cd8e73fefa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:49:59.542996 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.542971 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-workload-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:49:59.543053 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.542997 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee01b31-2432-4494-a93c-20cd8e73fefa-istiod-ca-cert\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:49:59.543053 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.543018 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-data" (OuterVolumeSpecName: "istio-data") pod "aee01b31-2432-4494-a93c-20cd8e73fefa" (UID: "aee01b31-2432-4494-a93c-20cd8e73fefa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:49:59.543053 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.543033 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aee01b31-2432-4494-a93c-20cd8e73fefa" (UID: "aee01b31-2432-4494-a93c-20cd8e73fefa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:49:59.543193 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.543078 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aee01b31-2432-4494-a93c-20cd8e73fefa" (UID: "aee01b31-2432-4494-a93c-20cd8e73fefa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:49:59.544495 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.544470 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aee01b31-2432-4494-a93c-20cd8e73fefa" (UID: "aee01b31-2432-4494-a93c-20cd8e73fefa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 02:49:59.544666 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.544645 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-token" (OuterVolumeSpecName: "istio-token") pod "aee01b31-2432-4494-a93c-20cd8e73fefa" (UID: "aee01b31-2432-4494-a93c-20cd8e73fefa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:49:59.544979 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.544958 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee01b31-2432-4494-a93c-20cd8e73fefa-kube-api-access-kd6sz" (OuterVolumeSpecName: "kube-api-access-kd6sz") pod "aee01b31-2432-4494-a93c-20cd8e73fefa" (UID: "aee01b31-2432-4494-a93c-20cd8e73fefa"). InnerVolumeSpecName "kube-api-access-kd6sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:49:59.545041 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.544960 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aee01b31-2432-4494-a93c-20cd8e73fefa" (UID: "aee01b31-2432-4494-a93c-20cd8e73fefa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:49:59.644104 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.644066 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-podinfo\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:49:59.644104 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.644090 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-envoy\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:49:59.644104 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.644098 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd6sz\" (UniqueName: \"kubernetes.io/projected/aee01b31-2432-4494-a93c-20cd8e73fefa-kube-api-access-kd6sz\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:49:59.644104 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.644108 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-credential-socket\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:49:59.644104 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.644117 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-data\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:49:59.644398 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.644125 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee01b31-2432-4494-a93c-20cd8e73fefa-workload-socket\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:49:59.644398 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:49:59.644133 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee01b31-2432-4494-a93c-20cd8e73fefa-istio-token\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:50:00.459889 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:00.459857 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" Sep 09 02:50:00.459889 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:00.459883 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8" event={"ID":"aee01b31-2432-4494-a93c-20cd8e73fefa","Type":"ContainerDied","Data":"976506c7f09aceee3a4ee3e82b66ade30d920b784151a05608e2005255dc1946"} Sep 09 02:50:00.460346 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:00.459949 2577 scope.go:117] "RemoveContainer" containerID="4cb7cc8052fc295ade6473ab98b1575716f29045e34c41435f8a766cf53b7ec1" Sep 09 02:50:00.478608 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:00.478576 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8"] Sep 09 02:50:00.482205 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:00.482170 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b18109-openshift-default-758cc57864-s86q8"] Sep 09 02:50:02.310550 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:02.310515 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aee01b31-2432-4494-a93c-20cd8e73fefa" path="/var/lib/kubelet/pods/aee01b31-2432-4494-a93c-20cd8e73fefa/volumes" Sep 09 02:50:04.856315 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:04.856279 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:05.432530 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.432496 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df"] Sep 09 02:50:05.432811 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.432783 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" podUID="974e5b86-02e4-42b9-a1ad-ab85414ec680" containerName="maas-api" containerID="cri-o://ab44afa535d73980dd38b6f2308a5d440aa5e7ddfb2b4154a035fe1a79712774" gracePeriod=30 Sep 09 02:50:05.486576 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.486542 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk"] Sep 09 02:50:05.487199 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.486976 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk" podUID="717f071e-8c75-454f-96f4-8e42ca69738b" containerName="payload-pre-processing" containerID="cri-o://1104c6ef0d10a0689656755173c04647ae5726a91fc9c295b676add948f99733" gracePeriod=30 Sep 09 02:50:05.575692 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.575661 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:05.665243 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.665211 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:05.732289 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.732263 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:05.734298 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.734277 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" Sep 09 02:50:05.763487 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.763465 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk" Sep 09 02:50:05.802411 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.802379 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/974e5b86-02e4-42b9-a1ad-ab85414ec680-maas-api-tls\") pod \"974e5b86-02e4-42b9-a1ad-ab85414ec680\" (UID: \"974e5b86-02e4-42b9-a1ad-ab85414ec680\") " Sep 09 02:50:05.802560 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.802426 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccr99\" (UniqueName: \"kubernetes.io/projected/974e5b86-02e4-42b9-a1ad-ab85414ec680-kube-api-access-ccr99\") pod \"974e5b86-02e4-42b9-a1ad-ab85414ec680\" (UID: \"974e5b86-02e4-42b9-a1ad-ab85414ec680\") " Sep 09 02:50:05.802560 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.802476 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/974e5b86-02e4-42b9-a1ad-ab85414ec680-openshift-service-ca\") pod \"974e5b86-02e4-42b9-a1ad-ab85414ec680\" (UID: \"974e5b86-02e4-42b9-a1ad-ab85414ec680\") " Sep 09 02:50:05.802560 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.802498 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/974e5b86-02e4-42b9-a1ad-ab85414ec680-metrics-certs\") pod \"974e5b86-02e4-42b9-a1ad-ab85414ec680\" (UID: \"974e5b86-02e4-42b9-a1ad-ab85414ec680\") " Sep 09 02:50:05.802934 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.802905 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974e5b86-02e4-42b9-a1ad-ab85414ec680-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "974e5b86-02e4-42b9-a1ad-ab85414ec680" (UID: "974e5b86-02e4-42b9-a1ad-ab85414ec680"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:50:05.804593 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.804558 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/974e5b86-02e4-42b9-a1ad-ab85414ec680-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "974e5b86-02e4-42b9-a1ad-ab85414ec680" (UID: "974e5b86-02e4-42b9-a1ad-ab85414ec680"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:50:05.804675 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.804636 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/974e5b86-02e4-42b9-a1ad-ab85414ec680-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "974e5b86-02e4-42b9-a1ad-ab85414ec680" (UID: "974e5b86-02e4-42b9-a1ad-ab85414ec680"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:50:05.804675 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.804646 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/974e5b86-02e4-42b9-a1ad-ab85414ec680-kube-api-access-ccr99" (OuterVolumeSpecName: "kube-api-access-ccr99") pod "974e5b86-02e4-42b9-a1ad-ab85414ec680" (UID: "974e5b86-02e4-42b9-a1ad-ab85414ec680"). InnerVolumeSpecName "kube-api-access-ccr99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:50:05.903447 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.903416 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9zzx\" (UniqueName: \"kubernetes.io/projected/717f071e-8c75-454f-96f4-8e42ca69738b-kube-api-access-h9zzx\") pod \"717f071e-8c75-454f-96f4-8e42ca69738b\" (UID: \"717f071e-8c75-454f-96f4-8e42ca69738b\") " Sep 09 02:50:05.903936 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.903458 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/717f071e-8c75-454f-96f4-8e42ca69738b-plugins-config-volume\") pod \"717f071e-8c75-454f-96f4-8e42ca69738b\" (UID: \"717f071e-8c75-454f-96f4-8e42ca69738b\") " Sep 09 02:50:05.903936 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.903771 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/974e5b86-02e4-42b9-a1ad-ab85414ec680-maas-api-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:50:05.903936 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.903791 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccr99\" (UniqueName: \"kubernetes.io/projected/974e5b86-02e4-42b9-a1ad-ab85414ec680-kube-api-access-ccr99\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:50:05.903936 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.903805 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/974e5b86-02e4-42b9-a1ad-ab85414ec680-openshift-service-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:50:05.903936 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.903844 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/974e5b86-02e4-42b9-a1ad-ab85414ec680-metrics-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:50:05.903936 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.903846 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/717f071e-8c75-454f-96f4-8e42ca69738b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "717f071e-8c75-454f-96f4-8e42ca69738b" (UID: "717f071e-8c75-454f-96f4-8e42ca69738b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:50:05.905537 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:05.905515 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717f071e-8c75-454f-96f4-8e42ca69738b-kube-api-access-h9zzx" (OuterVolumeSpecName: "kube-api-access-h9zzx") pod "717f071e-8c75-454f-96f4-8e42ca69738b" (UID: "717f071e-8c75-454f-96f4-8e42ca69738b"). InnerVolumeSpecName "kube-api-access-h9zzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:50:06.005033 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.004957 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9zzx\" (UniqueName: \"kubernetes.io/projected/717f071e-8c75-454f-96f4-8e42ca69738b-kube-api-access-h9zzx\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:50:06.005033 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.004983 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/717f071e-8c75-454f-96f4-8e42ca69738b-plugins-config-volume\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:50:06.107601 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.107569 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:06.207749 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.207713 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:06.496928 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.496890 2577 generic.go:356] "Generic (PLEG): container finished" podID="717f071e-8c75-454f-96f4-8e42ca69738b" containerID="1104c6ef0d10a0689656755173c04647ae5726a91fc9c295b676add948f99733" exitCode=0 Sep 09 02:50:06.497160 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.496944 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk" Sep 09 02:50:06.497160 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.496969 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk" event={"ID":"717f071e-8c75-454f-96f4-8e42ca69738b","Type":"ContainerDied","Data":"1104c6ef0d10a0689656755173c04647ae5726a91fc9c295b676add948f99733"} Sep 09 02:50:06.497160 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.497031 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk" event={"ID":"717f071e-8c75-454f-96f4-8e42ca69738b","Type":"ContainerDied","Data":"375d869e3aedf54a0de466776438cc59798b7b615d36f93ced8d509a40ac3960"} Sep 09 02:50:06.497160 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.497055 2577 scope.go:117] "RemoveContainer" containerID="1104c6ef0d10a0689656755173c04647ae5726a91fc9c295b676add948f99733" Sep 09 02:50:06.498628 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.498595 2577 generic.go:356] "Generic (PLEG): container finished" podID="974e5b86-02e4-42b9-a1ad-ab85414ec680" containerID="ab44afa535d73980dd38b6f2308a5d440aa5e7ddfb2b4154a035fe1a79712774" exitCode=0 Sep 09 02:50:06.498754 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.498631 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" event={"ID":"974e5b86-02e4-42b9-a1ad-ab85414ec680","Type":"ContainerDied","Data":"ab44afa535d73980dd38b6f2308a5d440aa5e7ddfb2b4154a035fe1a79712774"} Sep 09 02:50:06.498754 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.498678 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" event={"ID":"974e5b86-02e4-42b9-a1ad-ab85414ec680","Type":"ContainerDied","Data":"a55545352f8bbc0a45871af99216f9c352fe96d087bb1b17b8ab491345e62c73"} Sep 09 02:50:06.498754 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.498690 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df" Sep 09 02:50:06.508417 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.508384 2577 scope.go:117] "RemoveContainer" containerID="1104c6ef0d10a0689656755173c04647ae5726a91fc9c295b676add948f99733" Sep 09 02:50:06.508754 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:50:06.508731 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1104c6ef0d10a0689656755173c04647ae5726a91fc9c295b676add948f99733\": container with ID starting with 1104c6ef0d10a0689656755173c04647ae5726a91fc9c295b676add948f99733 not found: ID does not exist" containerID="1104c6ef0d10a0689656755173c04647ae5726a91fc9c295b676add948f99733" Sep 09 02:50:06.508848 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.508763 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1104c6ef0d10a0689656755173c04647ae5726a91fc9c295b676add948f99733"} err="failed to get container status \"1104c6ef0d10a0689656755173c04647ae5726a91fc9c295b676add948f99733\": rpc error: code = NotFound desc = could not find container \"1104c6ef0d10a0689656755173c04647ae5726a91fc9c295b676add948f99733\": container with ID starting with 1104c6ef0d10a0689656755173c04647ae5726a91fc9c295b676add948f99733 not found: ID does not exist" Sep 09 02:50:06.508848 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.508785 2577 scope.go:117] "RemoveContainer" containerID="ab44afa535d73980dd38b6f2308a5d440aa5e7ddfb2b4154a035fe1a79712774" Sep 09 02:50:06.518387 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.518361 2577 scope.go:117] "RemoveContainer" containerID="ab44afa535d73980dd38b6f2308a5d440aa5e7ddfb2b4154a035fe1a79712774" Sep 09 02:50:06.518735 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:50:06.518708 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab44afa535d73980dd38b6f2308a5d440aa5e7ddfb2b4154a035fe1a79712774\": container with ID starting with ab44afa535d73980dd38b6f2308a5d440aa5e7ddfb2b4154a035fe1a79712774 not found: ID does not exist" containerID="ab44afa535d73980dd38b6f2308a5d440aa5e7ddfb2b4154a035fe1a79712774" Sep 09 02:50:06.518810 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.518743 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab44afa535d73980dd38b6f2308a5d440aa5e7ddfb2b4154a035fe1a79712774"} err="failed to get container status \"ab44afa535d73980dd38b6f2308a5d440aa5e7ddfb2b4154a035fe1a79712774\": rpc error: code = NotFound desc = could not find container \"ab44afa535d73980dd38b6f2308a5d440aa5e7ddfb2b4154a035fe1a79712774\": container with ID starting with ab44afa535d73980dd38b6f2308a5d440aa5e7ddfb2b4154a035fe1a79712774 not found: ID does not exist" Sep 09 02:50:06.520265 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.520243 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk"] Sep 09 02:50:06.526047 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.526021 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6b11a2-6f6dd65b94-bxgbk"] Sep 09 02:50:06.536855 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.536826 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df"] Sep 09 02:50:06.540681 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.540655 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6b11a2-6cdb59cddb-rt4df"] Sep 09 02:50:06.658337 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.658307 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447"] Sep 09 02:50:06.658542 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.658514 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" podUID="23f1c5f5-124d-4519-9bdd-6fd7adefa0d7" containerName="maas-api" containerID="cri-o://650d9038e64ad5d7b85084798c3611f64517e15f4c050a2776a0e70e314be33d" gracePeriod=30 Sep 09 02:50:06.745114 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.745076 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:06.836515 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.836473 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:06.905190 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.905163 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" Sep 09 02:50:06.912756 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.912690 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:06.980915 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:06.980879 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:07.013085 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:07.013055 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7pdd\" (UniqueName: \"kubernetes.io/projected/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-kube-api-access-c7pdd\") pod \"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7\" (UID: \"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7\") " Sep 09 02:50:07.013085 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:07.013085 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-maas-api-tls\") pod \"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7\" (UID: \"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7\") " Sep 09 02:50:07.013302 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:07.013132 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-metrics-certs\") pod \"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7\" (UID: \"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7\") " Sep 09 02:50:07.013302 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:07.013192 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-openshift-service-ca\") pod \"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7\" (UID: \"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7\") " Sep 09 02:50:07.013611 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:07.013576 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "23f1c5f5-124d-4519-9bdd-6fd7adefa0d7" (UID: "23f1c5f5-124d-4519-9bdd-6fd7adefa0d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:50:07.015288 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:07.015260 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "23f1c5f5-124d-4519-9bdd-6fd7adefa0d7" (UID: "23f1c5f5-124d-4519-9bdd-6fd7adefa0d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:50:07.015399 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:07.015281 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "23f1c5f5-124d-4519-9bdd-6fd7adefa0d7" (UID: "23f1c5f5-124d-4519-9bdd-6fd7adefa0d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:50:07.015399 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:07.015317 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-kube-api-access-c7pdd" (OuterVolumeSpecName: "kube-api-access-c7pdd") pod "23f1c5f5-124d-4519-9bdd-6fd7adefa0d7" (UID: "23f1c5f5-124d-4519-9bdd-6fd7adefa0d7"). InnerVolumeSpecName "kube-api-access-c7pdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:50:07.113993 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:07.113960 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-metrics-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:50:07.113993 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:07.113987 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-openshift-service-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:50:07.113993 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:07.113997 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7pdd\" (UniqueName: \"kubernetes.io/projected/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-kube-api-access-c7pdd\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:50:07.114223 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:07.114006 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7-maas-api-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:50:07.382769 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:07.382739 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:07.513524 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:07.513488 2577 generic.go:356] "Generic (PLEG): container finished" podID="23f1c5f5-124d-4519-9bdd-6fd7adefa0d7" containerID="650d9038e64ad5d7b85084798c3611f64517e15f4c050a2776a0e70e314be33d" exitCode=0 Sep 09 02:50:07.513718 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:07.513542 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" Sep 09 02:50:07.513718 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:07.513547 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" event={"ID":"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7","Type":"ContainerDied","Data":"650d9038e64ad5d7b85084798c3611f64517e15f4c050a2776a0e70e314be33d"} Sep 09 02:50:07.513718 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:07.513688 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447" event={"ID":"23f1c5f5-124d-4519-9bdd-6fd7adefa0d7","Type":"ContainerDied","Data":"1551e02b4bb15c61d182de62884cb356a008fce59b27209f408d62eb4c59eab4"} Sep 09 02:50:07.513718 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:07.513718 2577 scope.go:117] "RemoveContainer" containerID="650d9038e64ad5d7b85084798c3611f64517e15f4c050a2776a0e70e314be33d" Sep 09 02:50:07.522804 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:07.522784 2577 scope.go:117] "RemoveContainer" containerID="650d9038e64ad5d7b85084798c3611f64517e15f4c050a2776a0e70e314be33d" Sep 09 02:50:07.523146 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:50:07.523126 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650d9038e64ad5d7b85084798c3611f64517e15f4c050a2776a0e70e314be33d\": container with ID starting with 650d9038e64ad5d7b85084798c3611f64517e15f4c050a2776a0e70e314be33d not found: ID does not exist" containerID="650d9038e64ad5d7b85084798c3611f64517e15f4c050a2776a0e70e314be33d" Sep 09 02:50:07.523194 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:07.523154 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650d9038e64ad5d7b85084798c3611f64517e15f4c050a2776a0e70e314be33d"} err="failed to get container status \"650d9038e64ad5d7b85084798c3611f64517e15f4c050a2776a0e70e314be33d\": rpc error: code = NotFound desc = could not find container \"650d9038e64ad5d7b85084798c3611f64517e15f4c050a2776a0e70e314be33d\": container with ID starting with 650d9038e64ad5d7b85084798c3611f64517e15f4c050a2776a0e70e314be33d not found: ID does not exist" Sep 09 02:50:07.534645 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:07.534496 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447"] Sep 09 02:50:07.536499 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:07.536477 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81f1254e-6bdc6c6688-wt447"] Sep 09 02:50:08.309723 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:08.309690 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23f1c5f5-124d-4519-9bdd-6fd7adefa0d7" path="/var/lib/kubelet/pods/23f1c5f5-124d-4519-9bdd-6fd7adefa0d7/volumes" Sep 09 02:50:08.310148 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:08.310119 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="717f071e-8c75-454f-96f4-8e42ca69738b" path="/var/lib/kubelet/pods/717f071e-8c75-454f-96f4-8e42ca69738b/volumes" Sep 09 02:50:08.310471 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:08.310458 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="974e5b86-02e4-42b9-a1ad-ab85414ec680" path="/var/lib/kubelet/pods/974e5b86-02e4-42b9-a1ad-ab85414ec680/volumes" Sep 09 02:50:09.361672 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:09.361638 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:12.092113 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:12.092075 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:12.157611 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:12.157574 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:12.365804 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:12.365719 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/2.log" Sep 09 02:50:12.366151 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:12.366135 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/2.log" Sep 09 02:50:12.410171 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:12.410139 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:12.921907 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:12.921874 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:13.223761 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.223676 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f"] Sep 09 02:50:13.224226 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.224116 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="717f071e-8c75-454f-96f4-8e42ca69738b" containerName="payload-pre-processing" Sep 09 02:50:13.224226 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.224129 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="717f071e-8c75-454f-96f4-8e42ca69738b" containerName="payload-pre-processing" Sep 09 02:50:13.224226 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.224148 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aee01b31-2432-4494-a93c-20cd8e73fefa" containerName="istio-proxy" Sep 09 02:50:13.224226 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.224153 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aee01b31-2432-4494-a93c-20cd8e73fefa" containerName="istio-proxy" Sep 09 02:50:13.224226 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.224160 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb7aebac-c330-497e-a0bd-ed44095a0486" containerName="maas-api" Sep 09 02:50:13.224226 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.224166 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb7aebac-c330-497e-a0bd-ed44095a0486" containerName="maas-api" Sep 09 02:50:13.224226 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.224175 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e90514e8-9cdb-40c2-9deb-df46033e98a4" containerName="istio-proxy" Sep 09 02:50:13.224226 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.224180 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e90514e8-9cdb-40c2-9deb-df46033e98a4" containerName="istio-proxy" Sep 09 02:50:13.224226 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.224189 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="974e5b86-02e4-42b9-a1ad-ab85414ec680" containerName="maas-api" Sep 09 02:50:13.224226 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.224194 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="974e5b86-02e4-42b9-a1ad-ab85414ec680" containerName="maas-api" Sep 09 02:50:13.224226 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.224200 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23f1c5f5-124d-4519-9bdd-6fd7adefa0d7" containerName="maas-api" Sep 09 02:50:13.224226 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.224205 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="23f1c5f5-124d-4519-9bdd-6fd7adefa0d7" containerName="maas-api" Sep 09 02:50:13.224226 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.224217 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aee01b31-2432-4494-a93c-20cd8e73fefa" containerName="istio-proxy" Sep 09 02:50:13.224226 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.224222 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aee01b31-2432-4494-a93c-20cd8e73fefa" containerName="istio-proxy" Sep 09 02:50:13.224889 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.224279 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="aee01b31-2432-4494-a93c-20cd8e73fefa" containerName="istio-proxy" Sep 09 02:50:13.224889 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.224287 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb7aebac-c330-497e-a0bd-ed44095a0486" containerName="maas-api" Sep 09 02:50:13.224889 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.224292 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="aee01b31-2432-4494-a93c-20cd8e73fefa" containerName="istio-proxy" Sep 09 02:50:13.224889 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.224298 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="23f1c5f5-124d-4519-9bdd-6fd7adefa0d7" containerName="maas-api" Sep 09 02:50:13.224889 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.224306 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="974e5b86-02e4-42b9-a1ad-ab85414ec680" containerName="maas-api" Sep 09 02:50:13.224889 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.224311 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="717f071e-8c75-454f-96f4-8e42ca69738b" containerName="payload-pre-processing" Sep 09 02:50:13.224889 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.224316 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e90514e8-9cdb-40c2-9deb-df46033e98a4" containerName="istio-proxy" Sep 09 02:50:13.229144 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.229122 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.231791 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.231768 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-56dbc0-openshift-default-dockercfg-nmvs8\"" Sep 09 02:50:13.240150 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.240127 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f"] Sep 09 02:50:13.258194 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.258171 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:13.372808 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.372772 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-workload-socket\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.373020 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.372882 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-token\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.373020 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.372916 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-workload-certs\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.373020 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.372961 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-credential-socket\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.373020 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.372998 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddntv\" (UniqueName: \"kubernetes.io/projected/e4aeb107-020b-4588-943d-ce3f93f8d571-kube-api-access-ddntv\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.373198 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.373021 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-podinfo\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.373198 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.373075 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-data\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.373198 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.373143 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4aeb107-020b-4588-943d-ce3f93f8d571-istiod-ca-cert\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.373198 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.373167 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-envoy\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.458669 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.458633 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:13.474561 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.474494 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4aeb107-020b-4588-943d-ce3f93f8d571-istiod-ca-cert\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.474561 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.474528 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-envoy\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.474760 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.474718 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-workload-socket\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.474870 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.474847 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-token\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.475064 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.475041 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-workload-socket\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.475150 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.475101 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-workload-certs\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.475210 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.475167 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-credential-socket\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.475306 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.475276 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddntv\" (UniqueName: \"kubernetes.io/projected/e4aeb107-020b-4588-943d-ce3f93f8d571-kube-api-access-ddntv\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.475306 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.475296 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4aeb107-020b-4588-943d-ce3f93f8d571-istiod-ca-cert\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.475485 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.475326 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-podinfo\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.475485 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.475362 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-data\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.475485 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.475413 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-workload-certs\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.475485 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.475455 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-credential-socket\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.475718 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.475682 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-data\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.476951 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.476928 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-envoy\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.477709 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.477679 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-podinfo\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.483099 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.483075 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-token\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.483202 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.483157 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddntv\" (UniqueName: \"kubernetes.io/projected/e4aeb107-020b-4588-943d-ce3f93f8d571-kube-api-access-ddntv\") pod \"e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.540920 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.540889 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:13.622948 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.622899 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:13.684434 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.684401 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:13.823458 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.823432 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f"] Sep 09 02:50:13.825414 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:50:13.825387 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4aeb107_020b_4588_943d_ce3f93f8d571.slice/crio-9de84e4e63930cf2bdee180ad4433440e5120bc8ac27c340ad8cb4882e51061b WatchSource:0}: Error finding container 9de84e4e63930cf2bdee180ad4433440e5120bc8ac27c340ad8cb4882e51061b: Status 404 returned error can't find the container with id 9de84e4e63930cf2bdee180ad4433440e5120bc8ac27c340ad8cb4882e51061b Sep 09 02:50:13.827590 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.827559 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:50:13.827679 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.827623 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:50:13.827679 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.827653 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:50:13.881618 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.881591 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v"] Sep 09 02:50:13.881879 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.881809 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" podUID="aea90141-6b57-4ddb-bba4-046d278fe07e" containerName="istio-proxy" containerID="cri-o://e6aa2e368b0fc2b5ce56b69e47811f1a1df546ed047c5f7518acceb64fae963b" gracePeriod=30 Sep 09 02:50:13.901916 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:13.901891 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:14.264708 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:14.264672 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:14.355191 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:14.355161 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:14.486251 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:14.486218 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:14.553858 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:14.553742 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" event={"ID":"e4aeb107-020b-4588-943d-ce3f93f8d571","Type":"ContainerStarted","Data":"86e89e73bf72f9a49bf615436dc7fe1679054c2b185bdce3615768f867d185b4"} Sep 09 02:50:14.553858 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:14.553808 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" event={"ID":"e4aeb107-020b-4588-943d-ce3f93f8d571","Type":"ContainerStarted","Data":"9de84e4e63930cf2bdee180ad4433440e5120bc8ac27c340ad8cb4882e51061b"} Sep 09 02:50:14.577718 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:14.577666 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" podStartSLOduration=1.5776493010000001 podStartE2EDuration="1.577649301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:50:14.575369257 +0000 UTC m=+1202.705611380" watchObservedRunningTime="2026-09-09 02:50:14.577649301 +0000 UTC m=+1202.707891396" Sep 09 02:50:15.541433 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:15.541398 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:15.546580 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:15.546553 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:15.559138 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:15.559116 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:15.559836 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:15.559796 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:16.492846 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:16.492787 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:16.590336 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:16.590300 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:17.714281 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:17.714248 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx"] Sep 09 02:50:17.718294 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:17.718269 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" Sep 09 02:50:17.720964 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:17.720945 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-56dbc0-serving-cert\"" Sep 09 02:50:17.731071 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:17.731048 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx"] Sep 09 02:50:17.750066 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:17.750041 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:17.819512 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:17.819479 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx\" (UID: \"b0bf842c-206e-40ca-9d32-602d6d3fcd9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" Sep 09 02:50:17.819692 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:17.819539 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx\" (UID: \"b0bf842c-206e-40ca-9d32-602d6d3fcd9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" Sep 09 02:50:17.819692 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:17.819578 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjgk5\" (UniqueName: \"kubernetes.io/projected/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-kube-api-access-qjgk5\") pod \"maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx\" (UID: \"b0bf842c-206e-40ca-9d32-602d6d3fcd9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" Sep 09 02:50:17.819692 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:17.819642 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx\" (UID: \"b0bf842c-206e-40ca-9d32-602d6d3fcd9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" Sep 09 02:50:17.850976 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:17.850934 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:17.920312 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:17.920269 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx\" (UID: \"b0bf842c-206e-40ca-9d32-602d6d3fcd9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" Sep 09 02:50:17.920312 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:17.920313 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjgk5\" (UniqueName: \"kubernetes.io/projected/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-kube-api-access-qjgk5\") pod \"maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx\" (UID: \"b0bf842c-206e-40ca-9d32-602d6d3fcd9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" Sep 09 02:50:17.920569 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:17.920350 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx\" (UID: \"b0bf842c-206e-40ca-9d32-602d6d3fcd9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" Sep 09 02:50:17.920634 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:17.920571 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx\" (UID: \"b0bf842c-206e-40ca-9d32-602d6d3fcd9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" Sep 09 02:50:17.920992 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:17.920967 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx\" (UID: \"b0bf842c-206e-40ca-9d32-602d6d3fcd9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" Sep 09 02:50:17.922869 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:17.922852 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx\" (UID: \"b0bf842c-206e-40ca-9d32-602d6d3fcd9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" Sep 09 02:50:17.922934 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:17.922891 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx\" (UID: \"b0bf842c-206e-40ca-9d32-602d6d3fcd9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" Sep 09 02:50:17.928928 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:17.928907 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjgk5\" (UniqueName: \"kubernetes.io/projected/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-kube-api-access-qjgk5\") pod \"maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx\" (UID: \"b0bf842c-206e-40ca-9d32-602d6d3fcd9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" Sep 09 02:50:17.950874 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:17.950839 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:18.031326 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:18.031233 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" Sep 09 02:50:18.248936 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:18.248893 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:18.307497 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:50:18.307465 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bf842c_206e_40ca_9d32_602d6d3fcd9b.slice/crio-d85accf11f88edb7f56cddc94a1c85ae16fa1e9726af025b4d4779b04ed69b3c WatchSource:0}: Error finding container d85accf11f88edb7f56cddc94a1c85ae16fa1e9726af025b4d4779b04ed69b3c: Status 404 returned error can't find the container with id d85accf11f88edb7f56cddc94a1c85ae16fa1e9726af025b4d4779b04ed69b3c Sep 09 02:50:18.310278 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:18.310258 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx"] Sep 09 02:50:18.346530 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:18.346499 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:18.577154 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:18.577069 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" event={"ID":"b0bf842c-206e-40ca-9d32-602d6d3fcd9b","Type":"ContainerStarted","Data":"d85accf11f88edb7f56cddc94a1c85ae16fa1e9726af025b4d4779b04ed69b3c"} Sep 09 02:50:18.605430 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:18.605391 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:19.145453 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.145419 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:50:19.239318 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.239283 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz4wh\" (UniqueName: \"kubernetes.io/projected/aea90141-6b57-4ddb-bba4-046d278fe07e-kube-api-access-dz4wh\") pod \"aea90141-6b57-4ddb-bba4-046d278fe07e\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " Sep 09 02:50:19.239474 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.239336 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-token\") pod \"aea90141-6b57-4ddb-bba4-046d278fe07e\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " Sep 09 02:50:19.239474 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.239381 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-workload-socket\") pod \"aea90141-6b57-4ddb-bba4-046d278fe07e\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " Sep 09 02:50:19.239474 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.239400 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-envoy\") pod \"aea90141-6b57-4ddb-bba4-046d278fe07e\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " Sep 09 02:50:19.239474 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.239427 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea90141-6b57-4ddb-bba4-046d278fe07e-istiod-ca-cert\") pod \"aea90141-6b57-4ddb-bba4-046d278fe07e\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " Sep 09 02:50:19.239474 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.239453 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-credential-socket\") pod \"aea90141-6b57-4ddb-bba4-046d278fe07e\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " Sep 09 02:50:19.239760 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.239509 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-podinfo\") pod \"aea90141-6b57-4ddb-bba4-046d278fe07e\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " Sep 09 02:50:19.239760 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.239542 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-data\") pod \"aea90141-6b57-4ddb-bba4-046d278fe07e\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " Sep 09 02:50:19.239760 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.239590 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-workload-certs\") pod \"aea90141-6b57-4ddb-bba4-046d278fe07e\" (UID: \"aea90141-6b57-4ddb-bba4-046d278fe07e\") " Sep 09 02:50:19.239760 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.239682 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aea90141-6b57-4ddb-bba4-046d278fe07e" (UID: "aea90141-6b57-4ddb-bba4-046d278fe07e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:50:19.239988 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.239850 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aea90141-6b57-4ddb-bba4-046d278fe07e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aea90141-6b57-4ddb-bba4-046d278fe07e" (UID: "aea90141-6b57-4ddb-bba4-046d278fe07e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:50:19.239988 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.239878 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aea90141-6b57-4ddb-bba4-046d278fe07e" (UID: "aea90141-6b57-4ddb-bba4-046d278fe07e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:50:19.240101 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.240010 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-workload-socket\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:50:19.240101 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.240033 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea90141-6b57-4ddb-bba4-046d278fe07e-istiod-ca-cert\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:50:19.240101 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.240049 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-workload-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:50:19.240101 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.240015 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-data" (OuterVolumeSpecName: "istio-data") pod "aea90141-6b57-4ddb-bba4-046d278fe07e" (UID: "aea90141-6b57-4ddb-bba4-046d278fe07e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:50:19.240278 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.240169 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aea90141-6b57-4ddb-bba4-046d278fe07e" (UID: "aea90141-6b57-4ddb-bba4-046d278fe07e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:50:19.241726 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.241694 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea90141-6b57-4ddb-bba4-046d278fe07e-kube-api-access-dz4wh" (OuterVolumeSpecName: "kube-api-access-dz4wh") pod "aea90141-6b57-4ddb-bba4-046d278fe07e" (UID: "aea90141-6b57-4ddb-bba4-046d278fe07e"). InnerVolumeSpecName "kube-api-access-dz4wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:50:19.241847 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.241757 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-token" (OuterVolumeSpecName: "istio-token") pod "aea90141-6b57-4ddb-bba4-046d278fe07e" (UID: "aea90141-6b57-4ddb-bba4-046d278fe07e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:50:19.242005 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.241978 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aea90141-6b57-4ddb-bba4-046d278fe07e" (UID: "aea90141-6b57-4ddb-bba4-046d278fe07e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:50:19.242134 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.242072 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aea90141-6b57-4ddb-bba4-046d278fe07e" (UID: "aea90141-6b57-4ddb-bba4-046d278fe07e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 02:50:19.341200 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.341167 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-credential-socket\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:50:19.341200 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.341196 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-podinfo\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:50:19.341200 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.341206 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-data\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:50:19.341427 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.341214 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz4wh\" (UniqueName: \"kubernetes.io/projected/aea90141-6b57-4ddb-bba4-046d278fe07e-kube-api-access-dz4wh\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:50:19.341427 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.341224 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-token\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:50:19.341427 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.341231 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea90141-6b57-4ddb-bba4-046d278fe07e-istio-envoy\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:50:19.584251 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.584210 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" event={"ID":"b0bf842c-206e-40ca-9d32-602d6d3fcd9b","Type":"ContainerStarted","Data":"72b15296738fb251ea9e5b65c675f9b1416663b4d4236252d5bedd4db4c08702"} Sep 09 02:50:19.584455 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.584283 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" Sep 09 02:50:19.586054 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.586027 2577 generic.go:356] "Generic (PLEG): container finished" podID="aea90141-6b57-4ddb-bba4-046d278fe07e" containerID="e6aa2e368b0fc2b5ce56b69e47811f1a1df546ed047c5f7518acceb64fae963b" exitCode=0 Sep 09 02:50:19.586151 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.586063 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" Sep 09 02:50:19.586151 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.586081 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" event={"ID":"aea90141-6b57-4ddb-bba4-046d278fe07e","Type":"ContainerDied","Data":"e6aa2e368b0fc2b5ce56b69e47811f1a1df546ed047c5f7518acceb64fae963b"} Sep 09 02:50:19.586151 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.586121 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v" event={"ID":"aea90141-6b57-4ddb-bba4-046d278fe07e","Type":"ContainerDied","Data":"4c665256eb83ddefeda3af25938705c2184c0cf55006a4fa9c6224bb3b084dfd"} Sep 09 02:50:19.586151 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.586143 2577 scope.go:117] "RemoveContainer" containerID="e6aa2e368b0fc2b5ce56b69e47811f1a1df546ed047c5f7518acceb64fae963b" Sep 09 02:50:19.595777 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.595755 2577 scope.go:117] "RemoveContainer" containerID="04ded24bdf1c99970e035c8f96d432ffbceede51c96666dbb641d2fd74e40319" Sep 09 02:50:19.604390 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.604370 2577 scope.go:117] "RemoveContainer" containerID="e6aa2e368b0fc2b5ce56b69e47811f1a1df546ed047c5f7518acceb64fae963b" Sep 09 02:50:19.604708 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:50:19.604689 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6aa2e368b0fc2b5ce56b69e47811f1a1df546ed047c5f7518acceb64fae963b\": container with ID starting with e6aa2e368b0fc2b5ce56b69e47811f1a1df546ed047c5f7518acceb64fae963b not found: ID does not exist" containerID="e6aa2e368b0fc2b5ce56b69e47811f1a1df546ed047c5f7518acceb64fae963b" Sep 09 02:50:19.604759 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.604719 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6aa2e368b0fc2b5ce56b69e47811f1a1df546ed047c5f7518acceb64fae963b"} err="failed to get container status \"e6aa2e368b0fc2b5ce56b69e47811f1a1df546ed047c5f7518acceb64fae963b\": rpc error: code = NotFound desc = could not find container \"e6aa2e368b0fc2b5ce56b69e47811f1a1df546ed047c5f7518acceb64fae963b\": container with ID starting with e6aa2e368b0fc2b5ce56b69e47811f1a1df546ed047c5f7518acceb64fae963b not found: ID does not exist" Sep 09 02:50:19.604759 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.604740 2577 scope.go:117] "RemoveContainer" containerID="04ded24bdf1c99970e035c8f96d432ffbceede51c96666dbb641d2fd74e40319" Sep 09 02:50:19.605072 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:50:19.605047 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ded24bdf1c99970e035c8f96d432ffbceede51c96666dbb641d2fd74e40319\": container with ID starting with 04ded24bdf1c99970e035c8f96d432ffbceede51c96666dbb641d2fd74e40319 not found: ID does not exist" containerID="04ded24bdf1c99970e035c8f96d432ffbceede51c96666dbb641d2fd74e40319" Sep 09 02:50:19.605152 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.605076 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ded24bdf1c99970e035c8f96d432ffbceede51c96666dbb641d2fd74e40319"} err="failed to get container status \"04ded24bdf1c99970e035c8f96d432ffbceede51c96666dbb641d2fd74e40319\": rpc error: code = NotFound desc = could not find container \"04ded24bdf1c99970e035c8f96d432ffbceede51c96666dbb641d2fd74e40319\": container with ID starting with 04ded24bdf1c99970e035c8f96d432ffbceede51c96666dbb641d2fd74e40319 not found: ID does not exist" Sep 09 02:50:19.607276 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.607228 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" podStartSLOduration=2.133802312 podStartE2EDuration="2.607214402s" totalImagesPullingTime="473.41209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:50:19.605631743 +0000 UTC m=+1207.735873839" watchObservedRunningTime="2026-09-09 02:50:19.607214402 +0000 UTC m=+1207.737456498" Sep 09 02:50:19.625564 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.625541 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v"] Sep 09 02:50:19.636477 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:19.636444 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-81f1254e-openshift-default-7459f7457b-jst8v"] Sep 09 02:50:20.310257 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:20.310227 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aea90141-6b57-4ddb-bba4-046d278fe07e" path="/var/lib/kubelet/pods/aea90141-6b57-4ddb-bba4-046d278fe07e/volumes" Sep 09 02:50:21.419363 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:21.419316 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:21.610485 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:21.610439 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:21.703618 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:21.703476 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:21.817122 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:21.817085 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:25.597551 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:25.597515 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" Sep 09 02:50:27.568792 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:27.568758 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:27.631533 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:27.631505 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:28.504923 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:28.504879 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:28.810517 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:28.810428 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:29.541157 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:29.541107 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:30.814841 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:30.814794 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": read tcp 10.133.0.2:39888->10.133.0.58:15021: read: connection reset by peer" start-of-body= Sep 09 02:50:30.815249 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:30.814879 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" podUID="e4aeb107-020b-4588-943d-ce3f93f8d571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": read tcp 10.133.0.2:39888->10.133.0.58:15021: read: connection reset by peer" Sep 09 02:50:31.647701 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:31.647663 2577 generic.go:356] "Generic (PLEG): container finished" podID="e4aeb107-020b-4588-943d-ce3f93f8d571" containerID="86e89e73bf72f9a49bf615436dc7fe1679054c2b185bdce3615768f867d185b4" exitCode=0 Sep 09 02:50:31.647906 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:31.647728 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" event={"ID":"e4aeb107-020b-4588-943d-ce3f93f8d571","Type":"ContainerDied","Data":"86e89e73bf72f9a49bf615436dc7fe1679054c2b185bdce3615768f867d185b4"} Sep 09 02:50:31.648220 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:31.648196 2577 scope.go:117] "RemoveContainer" containerID="86e89e73bf72f9a49bf615436dc7fe1679054c2b185bdce3615768f867d185b4" Sep 09 02:50:31.649333 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:31.649306 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:50:31.649394 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:31.649373 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:50:31.649429 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:31.649401 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Sep 09 02:50:31.748114 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:31.748075 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:31.846025 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:31.845997 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:32.658178 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:32.657202 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" event={"ID":"e4aeb107-020b-4588-943d-ce3f93f8d571","Type":"ContainerStarted","Data":"cddd455f2ee888d46a68b4ec7fbacf74246f4f9871a81ab67799ca79ad2d7610"} Sep 09 02:50:32.929563 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:32.929473 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9"] Sep 09 02:50:32.930262 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:32.930220 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aea90141-6b57-4ddb-bba4-046d278fe07e" containerName="istio-proxy" Sep 09 02:50:32.930447 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:32.930431 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aea90141-6b57-4ddb-bba4-046d278fe07e" containerName="istio-proxy" Sep 09 02:50:32.930615 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:32.930601 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aea90141-6b57-4ddb-bba4-046d278fe07e" containerName="istio-proxy" Sep 09 02:50:32.930732 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:32.930712 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aea90141-6b57-4ddb-bba4-046d278fe07e" containerName="istio-proxy" Sep 09 02:50:32.930891 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:32.930862 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="aea90141-6b57-4ddb-bba4-046d278fe07e" containerName="istio-proxy" Sep 09 02:50:32.931040 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:32.931014 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="aea90141-6b57-4ddb-bba4-046d278fe07e" containerName="istio-proxy" Sep 09 02:50:32.933441 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:32.933423 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" Sep 09 02:50:32.936085 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:32.936066 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-d838aa-serving-cert\"" Sep 09 02:50:32.946514 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:32.946469 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9"] Sep 09 02:50:32.973840 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:32.973317 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623fc653-8a55-4a7a-87c9-b5e949e147aa-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9\" (UID: \"623fc653-8a55-4a7a-87c9-b5e949e147aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" Sep 09 02:50:32.973840 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:32.973407 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8qsq\" (UniqueName: \"kubernetes.io/projected/623fc653-8a55-4a7a-87c9-b5e949e147aa-kube-api-access-w8qsq\") pod \"maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9\" (UID: \"623fc653-8a55-4a7a-87c9-b5e949e147aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" Sep 09 02:50:32.973840 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:32.973440 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/623fc653-8a55-4a7a-87c9-b5e949e147aa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9\" (UID: \"623fc653-8a55-4a7a-87c9-b5e949e147aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" Sep 09 02:50:32.973840 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:32.973536 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/623fc653-8a55-4a7a-87c9-b5e949e147aa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9\" (UID: \"623fc653-8a55-4a7a-87c9-b5e949e147aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" Sep 09 02:50:32.980785 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:32.980752 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:33.048476 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:33.048445 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:33.075085 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:33.075037 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8qsq\" (UniqueName: \"kubernetes.io/projected/623fc653-8a55-4a7a-87c9-b5e949e147aa-kube-api-access-w8qsq\") pod \"maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9\" (UID: \"623fc653-8a55-4a7a-87c9-b5e949e147aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" Sep 09 02:50:33.075304 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:33.075100 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/623fc653-8a55-4a7a-87c9-b5e949e147aa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9\" (UID: \"623fc653-8a55-4a7a-87c9-b5e949e147aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" Sep 09 02:50:33.075304 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:33.075200 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/623fc653-8a55-4a7a-87c9-b5e949e147aa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9\" (UID: \"623fc653-8a55-4a7a-87c9-b5e949e147aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" Sep 09 02:50:33.075304 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:33.075271 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623fc653-8a55-4a7a-87c9-b5e949e147aa-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9\" (UID: \"623fc653-8a55-4a7a-87c9-b5e949e147aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" Sep 09 02:50:33.076840 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:33.075936 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/623fc653-8a55-4a7a-87c9-b5e949e147aa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9\" (UID: \"623fc653-8a55-4a7a-87c9-b5e949e147aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" Sep 09 02:50:33.078611 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:33.078579 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623fc653-8a55-4a7a-87c9-b5e949e147aa-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9\" (UID: \"623fc653-8a55-4a7a-87c9-b5e949e147aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" Sep 09 02:50:33.079458 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:33.079438 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/623fc653-8a55-4a7a-87c9-b5e949e147aa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9\" (UID: \"623fc653-8a55-4a7a-87c9-b5e949e147aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" Sep 09 02:50:33.084193 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:33.084175 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8qsq\" (UniqueName: \"kubernetes.io/projected/623fc653-8a55-4a7a-87c9-b5e949e147aa-kube-api-access-w8qsq\") pod \"maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9\" (UID: \"623fc653-8a55-4a7a-87c9-b5e949e147aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" Sep 09 02:50:33.112953 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:33.112923 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:33.244727 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:33.244632 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" Sep 09 02:50:33.473840 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:33.473365 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:33.537715 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:33.537689 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9"] Sep 09 02:50:33.539796 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:50:33.539765 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623fc653_8a55_4a7a_87c9_b5e949e147aa.slice/crio-8907c00c452b16ea68896c00cba1bba3a57832abe3bf6603f283663b7fe30380 WatchSource:0}: Error finding container 8907c00c452b16ea68896c00cba1bba3a57832abe3bf6603f283663b7fe30380: Status 404 returned error can't find the container with id 8907c00c452b16ea68896c00cba1bba3a57832abe3bf6603f283663b7fe30380 Sep 09 02:50:33.541120 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:33.541097 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:33.541310 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:33.541290 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 09 02:50:33.541363 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:33.541339 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" podUID="e4aeb107-020b-4588-943d-ce3f93f8d571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 09 02:50:33.545473 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:33.545448 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:33.672843 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:33.672088 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" event={"ID":"623fc653-8a55-4a7a-87c9-b5e949e147aa","Type":"ContainerStarted","Data":"8907c00c452b16ea68896c00cba1bba3a57832abe3bf6603f283663b7fe30380"} Sep 09 02:50:34.541593 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:34.541492 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 09 02:50:34.541593 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:34.541565 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" podUID="e4aeb107-020b-4588-943d-ce3f93f8d571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 09 02:50:34.681633 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:34.681590 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" event={"ID":"623fc653-8a55-4a7a-87c9-b5e949e147aa","Type":"ContainerStarted","Data":"93aa8674cf5fc4b1bc03f34776e50580f6a398e32011b0a99e1c9eeb35aab928"} Sep 09 02:50:34.681871 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:34.681806 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" Sep 09 02:50:34.703647 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:34.703591 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" podStartSLOduration=2.088825831 podStartE2EDuration="2.703575279s" totalImagesPullingTime="614.749448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:50:34.701033293 +0000 UTC m=+1222.831275389" watchObservedRunningTime="2026-09-09 02:50:34.703575279 +0000 UTC m=+1222.833817375" Sep 09 02:50:35.542084 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:35.542041 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 09 02:50:35.542544 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:35.542104 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" podUID="e4aeb107-020b-4588-943d-ce3f93f8d571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 09 02:50:36.542177 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:36.542138 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 09 02:50:36.542586 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:36.542196 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" podUID="e4aeb107-020b-4588-943d-ce3f93f8d571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 09 02:50:37.541651 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:37.541614 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 09 02:50:37.541936 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:37.541671 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" podUID="e4aeb107-020b-4588-943d-ce3f93f8d571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 09 02:50:38.544785 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:38.544750 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:38.545191 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:38.545158 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:38.545756 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:38.545738 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 02:50:38.645314 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:38.645277 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:40.693145 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:40.693118 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" Sep 09 02:50:41.452872 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:41.452813 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:41.775432 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:41.775320 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:42.094571 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:42.094450 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:42.270364 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:42.270324 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:44.808322 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:44.808285 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:44.904253 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:44.904222 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:45.935192 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:45.935158 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb"] Sep 09 02:50:45.938173 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:45.938155 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" Sep 09 02:50:45.940736 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:45.940713 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-c173ad-serving-cert\"" Sep 09 02:50:45.948289 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:45.948266 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb"] Sep 09 02:50:46.012074 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.012025 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:46.111866 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.111801 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4d5ed5-0912-4bba-b926-4614ea6a0776-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb\" (UID: \"2f4d5ed5-0912-4bba-b926-4614ea6a0776\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" Sep 09 02:50:46.112050 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.111917 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f4d5ed5-0912-4bba-b926-4614ea6a0776-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb\" (UID: \"2f4d5ed5-0912-4bba-b926-4614ea6a0776\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" Sep 09 02:50:46.112050 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.111977 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxf5b\" (UniqueName: \"kubernetes.io/projected/2f4d5ed5-0912-4bba-b926-4614ea6a0776-kube-api-access-lxf5b\") pod \"maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb\" (UID: \"2f4d5ed5-0912-4bba-b926-4614ea6a0776\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" Sep 09 02:50:46.112050 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.112031 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4d5ed5-0912-4bba-b926-4614ea6a0776-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb\" (UID: \"2f4d5ed5-0912-4bba-b926-4614ea6a0776\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" Sep 09 02:50:46.112303 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.112282 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:46.212621 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.212534 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4d5ed5-0912-4bba-b926-4614ea6a0776-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb\" (UID: \"2f4d5ed5-0912-4bba-b926-4614ea6a0776\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" Sep 09 02:50:46.212621 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.212611 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4d5ed5-0912-4bba-b926-4614ea6a0776-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb\" (UID: \"2f4d5ed5-0912-4bba-b926-4614ea6a0776\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" Sep 09 02:50:46.212879 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.212643 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f4d5ed5-0912-4bba-b926-4614ea6a0776-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb\" (UID: \"2f4d5ed5-0912-4bba-b926-4614ea6a0776\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" Sep 09 02:50:46.212879 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.212676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxf5b\" (UniqueName: \"kubernetes.io/projected/2f4d5ed5-0912-4bba-b926-4614ea6a0776-kube-api-access-lxf5b\") pod \"maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb\" (UID: \"2f4d5ed5-0912-4bba-b926-4614ea6a0776\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" Sep 09 02:50:46.212879 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:50:46.212722 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-serving-cert: secret "maas-api-e2e-shared-b-w3-c173ad-serving-cert" not found Sep 09 02:50:46.212879 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:50:46.212828 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f4d5ed5-0912-4bba-b926-4614ea6a0776-maas-api-tls podName:2f4d5ed5-0912-4bba-b926-4614ea6a0776 nodeName:}" failed. No retries permitted until 2026-09-09 02:50:46.712788839 +0000 UTC m=+1234.843030934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2f4d5ed5-0912-4bba-b926-4614ea6a0776-maas-api-tls") pod "maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" (UID: "2f4d5ed5-0912-4bba-b926-4614ea6a0776") : secret "maas-api-e2e-shared-b-w3-c173ad-serving-cert" not found Sep 09 02:50:46.213305 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.213282 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4d5ed5-0912-4bba-b926-4614ea6a0776-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb\" (UID: \"2f4d5ed5-0912-4bba-b926-4614ea6a0776\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" Sep 09 02:50:46.215233 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.215213 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f4d5ed5-0912-4bba-b926-4614ea6a0776-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb\" (UID: \"2f4d5ed5-0912-4bba-b926-4614ea6a0776\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" Sep 09 02:50:46.224109 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.224087 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxf5b\" (UniqueName: \"kubernetes.io/projected/2f4d5ed5-0912-4bba-b926-4614ea6a0776-kube-api-access-lxf5b\") pod \"maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb\" (UID: \"2f4d5ed5-0912-4bba-b926-4614ea6a0776\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" Sep 09 02:50:46.470103 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.469991 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:46.567487 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.567456 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:46.634253 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.634219 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8"] Sep 09 02:50:46.637996 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.637967 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8" Sep 09 02:50:46.640540 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.640518 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-c173ad\"" Sep 09 02:50:46.640645 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.640549 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-c173ad-dockercfg-ngtx5\"" Sep 09 02:50:46.650670 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.650643 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8"] Sep 09 02:50:46.653021 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.652989 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:46.716673 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.716634 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4d5ed5-0912-4bba-b926-4614ea6a0776-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb\" (UID: \"2f4d5ed5-0912-4bba-b926-4614ea6a0776\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" Sep 09 02:50:46.716925 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.716723 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwklg\" (UniqueName: \"kubernetes.io/projected/d5be666a-6315-45f7-8593-f6474ac6435b-kube-api-access-dwklg\") pod \"payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8\" (UID: \"d5be666a-6315-45f7-8593-f6474ac6435b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8" Sep 09 02:50:46.716925 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.716779 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5be666a-6315-45f7-8593-f6474ac6435b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8\" (UID: \"d5be666a-6315-45f7-8593-f6474ac6435b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8" Sep 09 02:50:46.719109 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.719082 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4d5ed5-0912-4bba-b926-4614ea6a0776-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb\" (UID: \"2f4d5ed5-0912-4bba-b926-4614ea6a0776\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" Sep 09 02:50:46.818051 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.817945 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwklg\" (UniqueName: \"kubernetes.io/projected/d5be666a-6315-45f7-8593-f6474ac6435b-kube-api-access-dwklg\") pod \"payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8\" (UID: \"d5be666a-6315-45f7-8593-f6474ac6435b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8" Sep 09 02:50:46.818051 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.818041 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5be666a-6315-45f7-8593-f6474ac6435b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8\" (UID: \"d5be666a-6315-45f7-8593-f6474ac6435b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8" Sep 09 02:50:46.818756 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.818736 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5be666a-6315-45f7-8593-f6474ac6435b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8\" (UID: \"d5be666a-6315-45f7-8593-f6474ac6435b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8" Sep 09 02:50:46.825781 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.825754 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwklg\" (UniqueName: \"kubernetes.io/projected/d5be666a-6315-45f7-8593-f6474ac6435b-kube-api-access-dwklg\") pod \"payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8\" (UID: \"d5be666a-6315-45f7-8593-f6474ac6435b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8" Sep 09 02:50:46.849800 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.849767 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" Sep 09 02:50:46.956363 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:46.956332 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8" Sep 09 02:50:47.123285 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:47.123248 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb"] Sep 09 02:50:47.125481 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:50:47.125453 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4d5ed5_0912_4bba_b926_4614ea6a0776.slice/crio-77a8450c83a9b75915e5f80f7ed91af6bb05e517844d045f6dc2b07e64bc49db WatchSource:0}: Error finding container 77a8450c83a9b75915e5f80f7ed91af6bb05e517844d045f6dc2b07e64bc49db: Status 404 returned error can't find the container with id 77a8450c83a9b75915e5f80f7ed91af6bb05e517844d045f6dc2b07e64bc49db Sep 09 02:50:47.229061 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:47.229031 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8"] Sep 09 02:50:47.231294 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:50:47.231264 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5be666a_6315_45f7_8593_f6474ac6435b.slice/crio-3ba6be3b0ab5337181657974c2f1e5335893b57f3cd3a4fc702063c33df008fa WatchSource:0}: Error finding container 3ba6be3b0ab5337181657974c2f1e5335893b57f3cd3a4fc702063c33df008fa: Status 404 returned error can't find the container with id 3ba6be3b0ab5337181657974c2f1e5335893b57f3cd3a4fc702063c33df008fa Sep 09 02:50:47.748755 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:47.748714 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8" event={"ID":"d5be666a-6315-45f7-8593-f6474ac6435b","Type":"ContainerStarted","Data":"df472d3d01f11a50ccf95ddfcfe74c1a2217b894454c4317b7d4f0b782b1a6ee"} Sep 09 02:50:47.748989 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:47.748781 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8" event={"ID":"d5be666a-6315-45f7-8593-f6474ac6435b","Type":"ContainerStarted","Data":"3ba6be3b0ab5337181657974c2f1e5335893b57f3cd3a4fc702063c33df008fa"} Sep 09 02:50:47.748989 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:47.748834 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8" Sep 09 02:50:47.750366 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:47.750341 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" event={"ID":"2f4d5ed5-0912-4bba-b926-4614ea6a0776","Type":"ContainerStarted","Data":"ac5b9b6d7b4cda603fc6b554a8158412f883c69fb4cc407d1084fbd87ab81fb7"} Sep 09 02:50:47.750468 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:47.750381 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" event={"ID":"2f4d5ed5-0912-4bba-b926-4614ea6a0776","Type":"ContainerStarted","Data":"77a8450c83a9b75915e5f80f7ed91af6bb05e517844d045f6dc2b07e64bc49db"} Sep 09 02:50:47.750468 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:47.750410 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" Sep 09 02:50:47.766203 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:47.766152 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8" podStartSLOduration=1.7661356289999999 podStartE2EDuration="1.766135629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:50:47.765093302 +0000 UTC m=+1235.895335399" watchObservedRunningTime="2026-09-09 02:50:47.766135629 +0000 UTC m=+1235.896377738" Sep 09 02:50:47.783196 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:47.783151 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" podStartSLOduration=2.339290633 podStartE2EDuration="2.783137184s" totalImagesPullingTime="443.846551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:50:47.781109703 +0000 UTC m=+1235.911351799" watchObservedRunningTime="2026-09-09 02:50:47.783137184 +0000 UTC m=+1235.913379279" Sep 09 02:50:53.760753 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:53.760724 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" Sep 09 02:50:55.168007 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:55.167974 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:56.718603 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:56.718565 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:57.061500 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:57.061416 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:58.360234 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:58.360196 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:50:58.757969 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:50:58.757940 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8" Sep 09 02:51:00.929387 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:00.929352 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:00.999507 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:00.999474 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:02.036039 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.036004 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk"] Sep 09 02:51:02.038937 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.038910 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" Sep 09 02:51:02.042061 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.042042 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-f5aebb-serving-cert\"" Sep 09 02:51:02.053312 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.053286 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk"] Sep 09 02:51:02.063494 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.063465 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655f79a2-a584-44f2-a615-2cad39a3a608-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk\" (UID: \"655f79a2-a584-44f2-a615-2cad39a3a608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" Sep 09 02:51:02.063653 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.063506 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkwgd\" (UniqueName: \"kubernetes.io/projected/655f79a2-a584-44f2-a615-2cad39a3a608-kube-api-access-lkwgd\") pod \"maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk\" (UID: \"655f79a2-a584-44f2-a615-2cad39a3a608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" Sep 09 02:51:02.063653 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.063599 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655f79a2-a584-44f2-a615-2cad39a3a608-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk\" (UID: \"655f79a2-a584-44f2-a615-2cad39a3a608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" Sep 09 02:51:02.063653 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.063633 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655f79a2-a584-44f2-a615-2cad39a3a608-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk\" (UID: \"655f79a2-a584-44f2-a615-2cad39a3a608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" Sep 09 02:51:02.075935 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.075885 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:02.164616 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.164585 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655f79a2-a584-44f2-a615-2cad39a3a608-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk\" (UID: \"655f79a2-a584-44f2-a615-2cad39a3a608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" Sep 09 02:51:02.164796 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.164630 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655f79a2-a584-44f2-a615-2cad39a3a608-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk\" (UID: \"655f79a2-a584-44f2-a615-2cad39a3a608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" Sep 09 02:51:02.164796 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.164692 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655f79a2-a584-44f2-a615-2cad39a3a608-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk\" (UID: \"655f79a2-a584-44f2-a615-2cad39a3a608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" Sep 09 02:51:02.164796 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.164723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkwgd\" (UniqueName: \"kubernetes.io/projected/655f79a2-a584-44f2-a615-2cad39a3a608-kube-api-access-lkwgd\") pod \"maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk\" (UID: \"655f79a2-a584-44f2-a615-2cad39a3a608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" Sep 09 02:51:02.165009 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:51:02.164828 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-serving-cert: secret "maas-api-e2e-shared-b-w2-f5aebb-serving-cert" not found Sep 09 02:51:02.165009 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:51:02.164911 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/655f79a2-a584-44f2-a615-2cad39a3a608-maas-api-tls podName:655f79a2-a584-44f2-a615-2cad39a3a608 nodeName:}" failed. No retries permitted until 2026-09-09 02:51:02.664893557 +0000 UTC m=+1250.795135631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/655f79a2-a584-44f2-a615-2cad39a3a608-maas-api-tls") pod "maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" (UID: "655f79a2-a584-44f2-a615-2cad39a3a608") : secret "maas-api-e2e-shared-b-w2-f5aebb-serving-cert" not found Sep 09 02:51:02.165328 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.165309 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655f79a2-a584-44f2-a615-2cad39a3a608-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk\" (UID: \"655f79a2-a584-44f2-a615-2cad39a3a608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" Sep 09 02:51:02.167293 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.167264 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655f79a2-a584-44f2-a615-2cad39a3a608-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk\" (UID: \"655f79a2-a584-44f2-a615-2cad39a3a608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" Sep 09 02:51:02.176057 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.176031 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkwgd\" (UniqueName: \"kubernetes.io/projected/655f79a2-a584-44f2-a615-2cad39a3a608-kube-api-access-lkwgd\") pod \"maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk\" (UID: \"655f79a2-a584-44f2-a615-2cad39a3a608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" Sep 09 02:51:02.176894 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.176866 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:02.417868 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.417832 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:02.563863 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.563827 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:02.663335 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.663299 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:02.668782 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.668720 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655f79a2-a584-44f2-a615-2cad39a3a608-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk\" (UID: \"655f79a2-a584-44f2-a615-2cad39a3a608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" Sep 09 02:51:02.671312 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.671287 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655f79a2-a584-44f2-a615-2cad39a3a608-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk\" (UID: \"655f79a2-a584-44f2-a615-2cad39a3a608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" Sep 09 02:51:02.950584 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:02.950492 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" Sep 09 02:51:03.227809 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:03.227647 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk"] Sep 09 02:51:03.235145 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:51:03.235102 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655f79a2_a584_44f2_a615_2cad39a3a608.slice/crio-0d27e3c57212f67bca1bad3a9f23637774e51844ac962d9245dbdfd4997ca6dc WatchSource:0}: Error finding container 0d27e3c57212f67bca1bad3a9f23637774e51844ac962d9245dbdfd4997ca6dc: Status 404 returned error can't find the container with id 0d27e3c57212f67bca1bad3a9f23637774e51844ac962d9245dbdfd4997ca6dc Sep 09 02:51:03.844745 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:03.844709 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" event={"ID":"655f79a2-a584-44f2-a615-2cad39a3a608","Type":"ContainerStarted","Data":"858bee7071c9dcb3a71128974131fbb4b2404cfe479488dc872152ae98ac49b0"} Sep 09 02:51:03.844920 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:03.844773 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" event={"ID":"655f79a2-a584-44f2-a615-2cad39a3a608","Type":"ContainerStarted","Data":"0d27e3c57212f67bca1bad3a9f23637774e51844ac962d9245dbdfd4997ca6dc"} Sep 09 02:51:03.844920 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:03.844803 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" Sep 09 02:51:03.861589 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:03.861501 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" podStartSLOduration=1.320063655 podStartE2EDuration="1.861481779s" totalImagesPullingTime="541.418124ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:51:03.860596707 +0000 UTC m=+1251.990838804" watchObservedRunningTime="2026-09-09 02:51:03.861481779 +0000 UTC m=+1251.991723878" Sep 09 02:51:06.896147 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:06.896047 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:09.772578 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:09.772548 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:09.855505 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:09.855481 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" Sep 09 02:51:12.477887 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:12.477859 2577 scope.go:117] "RemoveContainer" containerID="ec6c0ca43a66a67af5d919cbd86c59156fa24e7a7d609d2c84c473e34ece5067" Sep 09 02:51:15.668149 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:15.668115 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:23.043800 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:23.043748 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:25.904160 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:25.904123 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:26.564773 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:26.564727 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7867-140487224616544"] Sep 09 02:51:26.570056 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:26.570028 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-140487224616544" Sep 09 02:51:26.573019 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:26.572990 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gnkf7\"" Sep 09 02:51:26.574879 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:26.574853 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7867-140487224616544"] Sep 09 02:51:26.706681 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:26.706640 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7c92\" (UniqueName: \"kubernetes.io/projected/3623e608-3a2f-4891-b23a-f0ad633fdb79-kube-api-access-t7c92\") pod \"test-curl-7867-140487224616544\" (UID: \"3623e608-3a2f-4891-b23a-f0ad633fdb79\") " pod="opendatahub/test-curl-7867-140487224616544" Sep 09 02:51:26.808156 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:26.808123 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7c92\" (UniqueName: \"kubernetes.io/projected/3623e608-3a2f-4891-b23a-f0ad633fdb79-kube-api-access-t7c92\") pod \"test-curl-7867-140487224616544\" (UID: \"3623e608-3a2f-4891-b23a-f0ad633fdb79\") " pod="opendatahub/test-curl-7867-140487224616544" Sep 09 02:51:26.816160 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:26.816096 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7c92\" (UniqueName: \"kubernetes.io/projected/3623e608-3a2f-4891-b23a-f0ad633fdb79-kube-api-access-t7c92\") pod \"test-curl-7867-140487224616544\" (UID: \"3623e608-3a2f-4891-b23a-f0ad633fdb79\") " pod="opendatahub/test-curl-7867-140487224616544" Sep 09 02:51:26.880359 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:26.880324 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-140487224616544" Sep 09 02:51:27.154549 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:27.154348 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7867-140487224616544"] Sep 09 02:51:27.157115 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:51:27.157088 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3623e608_3a2f_4891_b23a_f0ad633fdb79.slice/crio-7cf0f744164b520bac065616bc9b8e8ce4b4446a3e007558f6715e41e327c67b WatchSource:0}: Error finding container 7cf0f744164b520bac065616bc9b8e8ce4b4446a3e007558f6715e41e327c67b: Status 404 returned error can't find the container with id 7cf0f744164b520bac065616bc9b8e8ce4b4446a3e007558f6715e41e327c67b Sep 09 02:51:27.971470 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:27.971437 2577 generic.go:356] "Generic (PLEG): container finished" podID="3623e608-3a2f-4891-b23a-f0ad633fdb79" containerID="a2b04f72bc2525cf71a72f90713423a7e5125fbd058434a89ea1f092a8db3f7e" exitCode=0 Sep 09 02:51:27.971649 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:27.971510 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7867-140487224616544" event={"ID":"3623e608-3a2f-4891-b23a-f0ad633fdb79","Type":"ContainerDied","Data":"a2b04f72bc2525cf71a72f90713423a7e5125fbd058434a89ea1f092a8db3f7e"} Sep 09 02:51:27.971649 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:27.971542 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7867-140487224616544" event={"ID":"3623e608-3a2f-4891-b23a-f0ad633fdb79","Type":"ContainerStarted","Data":"7cf0f744164b520bac065616bc9b8e8ce4b4446a3e007558f6715e41e327c67b"} Sep 09 02:51:29.112198 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:29.112176 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-140487224616544" Sep 09 02:51:29.131837 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:29.131785 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7867-140487224616544_3623e608-3a2f-4891-b23a-f0ad633fdb79/test-curl-7867-140487224616544/0.log" Sep 09 02:51:29.152832 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:29.152782 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7867-140487224616544"] Sep 09 02:51:29.154961 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:29.154935 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7867-140487224616544"] Sep 09 02:51:29.229688 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:29.229605 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7c92\" (UniqueName: \"kubernetes.io/projected/3623e608-3a2f-4891-b23a-f0ad633fdb79-kube-api-access-t7c92\") pod \"3623e608-3a2f-4891-b23a-f0ad633fdb79\" (UID: \"3623e608-3a2f-4891-b23a-f0ad633fdb79\") " Sep 09 02:51:29.231686 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:29.231661 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3623e608-3a2f-4891-b23a-f0ad633fdb79-kube-api-access-t7c92" (OuterVolumeSpecName: "kube-api-access-t7c92") pod "3623e608-3a2f-4891-b23a-f0ad633fdb79" (UID: "3623e608-3a2f-4891-b23a-f0ad633fdb79"). InnerVolumeSpecName "kube-api-access-t7c92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:51:29.275155 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:29.275121 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7867-140487224616544"] Sep 09 02:51:29.275624 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:29.275606 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3623e608-3a2f-4891-b23a-f0ad633fdb79" containerName="test-curl-7867-140487224616544" Sep 09 02:51:29.275726 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:29.275627 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3623e608-3a2f-4891-b23a-f0ad633fdb79" containerName="test-curl-7867-140487224616544" Sep 09 02:51:29.275785 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:29.275768 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3623e608-3a2f-4891-b23a-f0ad633fdb79" containerName="test-curl-7867-140487224616544" Sep 09 02:51:29.279041 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:29.279021 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-140487224616544" Sep 09 02:51:29.287354 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:29.287326 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7867-140487224616544"] Sep 09 02:51:29.331091 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:29.331059 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7c92\" (UniqueName: \"kubernetes.io/projected/3623e608-3a2f-4891-b23a-f0ad633fdb79-kube-api-access-t7c92\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:51:29.431753 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:29.431710 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6spp\" (UniqueName: \"kubernetes.io/projected/f57fa101-1686-4319-8f39-7f15eb98d391-kube-api-access-f6spp\") pod \"test-curl-7867-140487224616544\" (UID: \"f57fa101-1686-4319-8f39-7f15eb98d391\") " pod="opendatahub/test-curl-7867-140487224616544" Sep 09 02:51:29.533334 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:29.533240 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6spp\" (UniqueName: \"kubernetes.io/projected/f57fa101-1686-4319-8f39-7f15eb98d391-kube-api-access-f6spp\") pod \"test-curl-7867-140487224616544\" (UID: \"f57fa101-1686-4319-8f39-7f15eb98d391\") " pod="opendatahub/test-curl-7867-140487224616544" Sep 09 02:51:29.542108 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:29.542076 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6spp\" (UniqueName: \"kubernetes.io/projected/f57fa101-1686-4319-8f39-7f15eb98d391-kube-api-access-f6spp\") pod \"test-curl-7867-140487224616544\" (UID: \"f57fa101-1686-4319-8f39-7f15eb98d391\") " pod="opendatahub/test-curl-7867-140487224616544" Sep 09 02:51:29.588770 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:29.588733 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-140487224616544" Sep 09 02:51:29.859269 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:29.859143 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7867-140487224616544"] Sep 09 02:51:29.861948 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:51:29.861919 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf57fa101_1686_4319_8f39_7f15eb98d391.slice/crio-ceb4d479a1e361049b9809182e2a4bad8c51de855bda0749b1f6b01127cd05e1 WatchSource:0}: Error finding container ceb4d479a1e361049b9809182e2a4bad8c51de855bda0749b1f6b01127cd05e1: Status 404 returned error can't find the container with id ceb4d479a1e361049b9809182e2a4bad8c51de855bda0749b1f6b01127cd05e1 Sep 09 02:51:29.983737 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:29.983703 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-140487224616544" Sep 09 02:51:29.983737 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:29.983722 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cf0f744164b520bac065616bc9b8e8ce4b4446a3e007558f6715e41e327c67b" Sep 09 02:51:29.985041 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:29.985012 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7867-140487224616544" event={"ID":"f57fa101-1686-4319-8f39-7f15eb98d391","Type":"ContainerStarted","Data":"ceb4d479a1e361049b9809182e2a4bad8c51de855bda0749b1f6b01127cd05e1"} Sep 09 02:51:29.995538 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:29.995505 2577 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7867-140487224616544" oldPodUID="3623e608-3a2f-4891-b23a-f0ad633fdb79" podUID="f57fa101-1686-4319-8f39-7f15eb98d391" Sep 09 02:51:30.311038 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:30.311003 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3623e608-3a2f-4891-b23a-f0ad633fdb79" path="/var/lib/kubelet/pods/3623e608-3a2f-4891-b23a-f0ad633fdb79/volumes" Sep 09 02:51:30.992206 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:30.992173 2577 generic.go:356] "Generic (PLEG): container finished" podID="f57fa101-1686-4319-8f39-7f15eb98d391" containerID="34be9881a26192b82e62de52e616997c3464c7c82013a77cbce297702034f871" exitCode=0 Sep 09 02:51:30.992396 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:30.992264 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7867-140487224616544" event={"ID":"f57fa101-1686-4319-8f39-7f15eb98d391","Type":"ContainerDied","Data":"34be9881a26192b82e62de52e616997c3464c7c82013a77cbce297702034f871"} Sep 09 02:51:32.131962 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:32.131940 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-140487224616544" Sep 09 02:51:32.151158 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:32.151131 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7867-140487224616544_f57fa101-1686-4319-8f39-7f15eb98d391/test-curl-7867-140487224616544/0.log" Sep 09 02:51:32.173057 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:32.173031 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7867-140487224616544"] Sep 09 02:51:32.176615 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:32.176591 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7867-140487224616544"] Sep 09 02:51:32.261733 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:32.261647 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6spp\" (UniqueName: \"kubernetes.io/projected/f57fa101-1686-4319-8f39-7f15eb98d391-kube-api-access-f6spp\") pod \"f57fa101-1686-4319-8f39-7f15eb98d391\" (UID: \"f57fa101-1686-4319-8f39-7f15eb98d391\") " Sep 09 02:51:32.263736 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:32.263710 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57fa101-1686-4319-8f39-7f15eb98d391-kube-api-access-f6spp" (OuterVolumeSpecName: "kube-api-access-f6spp") pod "f57fa101-1686-4319-8f39-7f15eb98d391" (UID: "f57fa101-1686-4319-8f39-7f15eb98d391"). InnerVolumeSpecName "kube-api-access-f6spp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:51:32.300350 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:32.300317 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7867-140487224617120"] Sep 09 02:51:32.300808 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:32.300794 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f57fa101-1686-4319-8f39-7f15eb98d391" containerName="test-curl-7867-140487224616544" Sep 09 02:51:32.300884 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:32.300809 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f57fa101-1686-4319-8f39-7f15eb98d391" containerName="test-curl-7867-140487224616544" Sep 09 02:51:32.300926 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:32.300899 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f57fa101-1686-4319-8f39-7f15eb98d391" containerName="test-curl-7867-140487224616544" Sep 09 02:51:32.308529 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:32.308497 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-140487224617120" Sep 09 02:51:32.313913 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:32.313885 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f57fa101-1686-4319-8f39-7f15eb98d391" path="/var/lib/kubelet/pods/f57fa101-1686-4319-8f39-7f15eb98d391/volumes" Sep 09 02:51:32.314262 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:32.314249 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7867-140487224617120"] Sep 09 02:51:32.363543 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:32.363510 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6spp\" (UniqueName: \"kubernetes.io/projected/f57fa101-1686-4319-8f39-7f15eb98d391-kube-api-access-f6spp\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:51:32.464535 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:32.464494 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdf2s\" (UniqueName: \"kubernetes.io/projected/6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237-kube-api-access-gdf2s\") pod \"test-curl-7867-140487224617120\" (UID: \"6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237\") " pod="opendatahub/test-curl-7867-140487224617120" Sep 09 02:51:32.565301 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:32.565217 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdf2s\" (UniqueName: \"kubernetes.io/projected/6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237-kube-api-access-gdf2s\") pod \"test-curl-7867-140487224617120\" (UID: \"6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237\") " pod="opendatahub/test-curl-7867-140487224617120" Sep 09 02:51:32.573908 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:32.573886 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdf2s\" (UniqueName: \"kubernetes.io/projected/6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237-kube-api-access-gdf2s\") pod \"test-curl-7867-140487224617120\" (UID: \"6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237\") " pod="opendatahub/test-curl-7867-140487224617120" Sep 09 02:51:32.620452 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:32.620420 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-140487224617120" Sep 09 02:51:32.889958 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:32.889936 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7867-140487224617120"] Sep 09 02:51:32.891532 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:51:32.891502 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a1dba9f_9d50_4c70_8aa0_cd54a1eb3237.slice/crio-afa80ebd3f0609bb8ce73853028d2682eae879dfb7676b0cade479c1bdefc544 WatchSource:0}: Error finding container afa80ebd3f0609bb8ce73853028d2682eae879dfb7676b0cade479c1bdefc544: Status 404 returned error can't find the container with id afa80ebd3f0609bb8ce73853028d2682eae879dfb7676b0cade479c1bdefc544 Sep 09 02:51:33.004788 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:33.004757 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-140487224616544" Sep 09 02:51:33.005005 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:33.004761 2577 scope.go:117] "RemoveContainer" containerID="34be9881a26192b82e62de52e616997c3464c7c82013a77cbce297702034f871" Sep 09 02:51:33.006087 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:33.006045 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7867-140487224617120" event={"ID":"6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237","Type":"ContainerStarted","Data":"afa80ebd3f0609bb8ce73853028d2682eae879dfb7676b0cade479c1bdefc544"} Sep 09 02:51:34.013378 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:34.013343 2577 generic.go:356] "Generic (PLEG): container finished" podID="6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237" containerID="69a63d9e6da9fbf96e3fa3bbf33ecc2a4b092423ce32ba0140120fcaf8bbfd72" exitCode=0 Sep 09 02:51:34.013790 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:34.013426 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7867-140487224617120" event={"ID":"6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237","Type":"ContainerDied","Data":"69a63d9e6da9fbf96e3fa3bbf33ecc2a4b092423ce32ba0140120fcaf8bbfd72"} Sep 09 02:51:35.150211 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:35.150182 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-140487224617120" Sep 09 02:51:35.169076 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:35.169047 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7867-140487224617120_6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237/test-curl-7867-140487224617120/0.log" Sep 09 02:51:35.192224 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:35.192182 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7867-140487224617120"] Sep 09 02:51:35.194298 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:35.194271 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7867-140487224617120"] Sep 09 02:51:35.293926 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:35.293796 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdf2s\" (UniqueName: \"kubernetes.io/projected/6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237-kube-api-access-gdf2s\") pod \"6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237\" (UID: \"6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237\") " Sep 09 02:51:35.296011 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:35.295980 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237-kube-api-access-gdf2s" (OuterVolumeSpecName: "kube-api-access-gdf2s") pod "6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237" (UID: "6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237"). InnerVolumeSpecName "kube-api-access-gdf2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:51:35.318069 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:35.317542 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7867-140487224617120"] Sep 09 02:51:35.318771 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:35.318701 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237" containerName="test-curl-7867-140487224617120" Sep 09 02:51:35.318771 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:35.318733 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237" containerName="test-curl-7867-140487224617120" Sep 09 02:51:35.319150 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:35.319101 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237" containerName="test-curl-7867-140487224617120" Sep 09 02:51:35.326428 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:35.326381 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-140487224617120" Sep 09 02:51:35.328359 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:35.328335 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7867-140487224617120"] Sep 09 02:51:35.395772 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:35.395731 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdf2s\" (UniqueName: \"kubernetes.io/projected/6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237-kube-api-access-gdf2s\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:51:35.497139 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:35.497096 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms8hw\" (UniqueName: \"kubernetes.io/projected/c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb-kube-api-access-ms8hw\") pod \"test-curl-7867-140487224617120\" (UID: \"c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb\") " pod="opendatahub/test-curl-7867-140487224617120" Sep 09 02:51:35.598355 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:35.598262 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms8hw\" (UniqueName: \"kubernetes.io/projected/c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb-kube-api-access-ms8hw\") pod \"test-curl-7867-140487224617120\" (UID: \"c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb\") " pod="opendatahub/test-curl-7867-140487224617120" Sep 09 02:51:35.606565 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:35.606531 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms8hw\" (UniqueName: \"kubernetes.io/projected/c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb-kube-api-access-ms8hw\") pod \"test-curl-7867-140487224617120\" (UID: \"c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb\") " pod="opendatahub/test-curl-7867-140487224617120" Sep 09 02:51:35.637478 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:35.637449 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-140487224617120" Sep 09 02:51:35.915004 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:35.909919 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7867-140487224617120"] Sep 09 02:51:36.024983 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:36.024952 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afa80ebd3f0609bb8ce73853028d2682eae879dfb7676b0cade479c1bdefc544" Sep 09 02:51:36.024983 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:36.024959 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-140487224617120" Sep 09 02:51:36.026322 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:36.026289 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7867-140487224617120" event={"ID":"c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb","Type":"ContainerStarted","Data":"9d5dd43b8b67671c5630deb1b75a5ac499f00d2ccecea2c9fca9a90a152712f5"} Sep 09 02:51:36.036636 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:36.036596 2577 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7867-140487224617120" oldPodUID="6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237" podUID="c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb" Sep 09 02:51:36.310025 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:36.309939 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237" path="/var/lib/kubelet/pods/6a1dba9f-9d50-4c70-8aa0-cd54a1eb3237/volumes" Sep 09 02:51:37.032883 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:37.032791 2577 generic.go:356] "Generic (PLEG): container finished" podID="c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb" containerID="5fbf77a46d4a61a0b3b4b70d15f2418a9b410111bc571c40ccf66ced21462cf1" exitCode=0 Sep 09 02:51:37.033033 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:37.032883 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7867-140487224617120" event={"ID":"c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb","Type":"ContainerDied","Data":"5fbf77a46d4a61a0b3b4b70d15f2418a9b410111bc571c40ccf66ced21462cf1"} Sep 09 02:51:38.174617 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:38.174588 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-140487224617120" Sep 09 02:51:38.194255 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:38.194231 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7867-140487224617120_c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb/test-curl-7867-140487224617120/0.log" Sep 09 02:51:38.214718 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:38.214681 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7867-140487224617120"] Sep 09 02:51:38.216544 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:38.216518 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7867-140487224617120"] Sep 09 02:51:38.220999 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:38.220981 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms8hw\" (UniqueName: \"kubernetes.io/projected/c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb-kube-api-access-ms8hw\") pod \"c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb\" (UID: \"c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb\") " Sep 09 02:51:38.222992 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:38.222969 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb-kube-api-access-ms8hw" (OuterVolumeSpecName: "kube-api-access-ms8hw") pod "c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb" (UID: "c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb"). InnerVolumeSpecName "kube-api-access-ms8hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:51:38.310904 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:38.310797 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb" path="/var/lib/kubelet/pods/c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb/volumes" Sep 09 02:51:38.322289 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:38.322267 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms8hw\" (UniqueName: \"kubernetes.io/projected/c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb-kube-api-access-ms8hw\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:51:38.852058 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:38.852005 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:39.044903 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:39.044878 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-140487224617120" Sep 09 02:51:39.045091 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:39.044936 2577 scope.go:117] "RemoveContainer" containerID="5fbf77a46d4a61a0b3b4b70d15f2418a9b410111bc571c40ccf66ced21462cf1" Sep 09 02:51:39.177153 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:39.177120 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:39.684910 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:39.684880 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:42.169554 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:42.169496 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:42.256238 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:42.256201 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:43.309269 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.309237 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb"] Sep 09 02:51:43.309718 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.309704 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb" containerName="test-curl-7867-140487224617120" Sep 09 02:51:43.309773 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.309719 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb" containerName="test-curl-7867-140487224617120" Sep 09 02:51:43.309848 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.309809 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9f2cd38-30a4-4eb6-8c68-e33f53ca73fb" containerName="test-curl-7867-140487224617120" Sep 09 02:51:43.313098 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.313004 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" Sep 09 02:51:43.315611 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.315588 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ad1d02af-serving-cert\"" Sep 09 02:51:43.325020 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.324990 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb"] Sep 09 02:51:43.341244 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.341184 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:43.374208 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.374168 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-maas-api-tls\") pod \"maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb\" (UID: \"17f2b36e-b365-4fcc-bbe5-a617bf1b4232\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" Sep 09 02:51:43.374409 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.374323 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7wgj\" (UniqueName: \"kubernetes.io/projected/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-kube-api-access-r7wgj\") pod \"maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb\" (UID: \"17f2b36e-b365-4fcc-bbe5-a617bf1b4232\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" Sep 09 02:51:43.374485 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.374466 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-openshift-service-ca\") pod \"maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb\" (UID: \"17f2b36e-b365-4fcc-bbe5-a617bf1b4232\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" Sep 09 02:51:43.374648 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.374626 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-metrics-certs\") pod \"maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb\" (UID: \"17f2b36e-b365-4fcc-bbe5-a617bf1b4232\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" Sep 09 02:51:43.441702 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.441660 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:43.476152 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.476115 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-openshift-service-ca\") pod \"maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb\" (UID: \"17f2b36e-b365-4fcc-bbe5-a617bf1b4232\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" Sep 09 02:51:43.476369 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.476274 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-metrics-certs\") pod \"maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb\" (UID: \"17f2b36e-b365-4fcc-bbe5-a617bf1b4232\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" Sep 09 02:51:43.476369 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.476307 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-maas-api-tls\") pod \"maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb\" (UID: \"17f2b36e-b365-4fcc-bbe5-a617bf1b4232\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" Sep 09 02:51:43.476369 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.476326 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7wgj\" (UniqueName: \"kubernetes.io/projected/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-kube-api-access-r7wgj\") pod \"maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb\" (UID: \"17f2b36e-b365-4fcc-bbe5-a617bf1b4232\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" Sep 09 02:51:43.476540 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:51:43.476424 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-serving-cert: secret "maas-api-e2e-mt-ad1d02af-serving-cert" not found Sep 09 02:51:43.476540 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:51:43.476492 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-maas-api-tls podName:17f2b36e-b365-4fcc-bbe5-a617bf1b4232 nodeName:}" failed. No retries permitted until 2026-09-09 02:51:43.976475248 +0000 UTC m=+1292.106717342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-maas-api-tls") pod "maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" (UID: "17f2b36e-b365-4fcc-bbe5-a617bf1b4232") : secret "maas-api-e2e-mt-ad1d02af-serving-cert" not found Sep 09 02:51:43.476927 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.476902 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-openshift-service-ca\") pod \"maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb\" (UID: \"17f2b36e-b365-4fcc-bbe5-a617bf1b4232\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" Sep 09 02:51:43.478706 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.478685 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-metrics-certs\") pod \"maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb\" (UID: \"17f2b36e-b365-4fcc-bbe5-a617bf1b4232\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" Sep 09 02:51:43.484673 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.484650 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7wgj\" (UniqueName: \"kubernetes.io/projected/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-kube-api-access-r7wgj\") pod \"maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb\" (UID: \"17f2b36e-b365-4fcc-bbe5-a617bf1b4232\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" Sep 09 02:51:43.670688 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.670649 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:43.835640 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.835595 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:43.928340 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.928247 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:43.981704 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.981666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-maas-api-tls\") pod \"maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb\" (UID: \"17f2b36e-b365-4fcc-bbe5-a617bf1b4232\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" Sep 09 02:51:43.984326 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:43.984293 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-maas-api-tls\") pod \"maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb\" (UID: \"17f2b36e-b365-4fcc-bbe5-a617bf1b4232\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" Sep 09 02:51:44.226794 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:44.226692 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" Sep 09 02:51:44.499844 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:44.499784 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb"] Sep 09 02:51:44.502137 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:51:44.502113 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f2b36e_b365_4fcc_bbe5_a617bf1b4232.slice/crio-94e3758da65b218d8a20b67fcf117f29f7bbc719b411f77a06cf39d2066bc5eb WatchSource:0}: Error finding container 94e3758da65b218d8a20b67fcf117f29f7bbc719b411f77a06cf39d2066bc5eb: Status 404 returned error can't find the container with id 94e3758da65b218d8a20b67fcf117f29f7bbc719b411f77a06cf39d2066bc5eb Sep 09 02:51:44.826896 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:51:44.826768 2577 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: initializing source docker://quay.io/opendatahub/maas-api@sha256:b9449be2bc36625833e33bec983da5e2d3bac9399f129efdd3fae5bf47fab5ea: unable to retrieve auth token: invalid username/password: authentication required; artifact err: provided artifact is a container image" image="quay.io/opendatahub/maas-api@sha256:b9449be2bc36625833e33bec983da5e2d3bac9399f129efdd3fae5bf47fab5ea" Sep 09 02:51:44.827065 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:51:44.826934 2577 kuberuntime_manager.go:1459] "Unhandled Error" err="container maas-api start failed in pod maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb_odh-ai-gateway-infra(17f2b36e-b365-4fcc-bbe5-a617bf1b4232): ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://quay.io/opendatahub/maas-api@sha256:b9449be2bc36625833e33bec983da5e2d3bac9399f129efdd3fae5bf47fab5ea: unable to retrieve auth token: invalid username/password: authentication required; artifact err: provided artifact is a container image" logger="UnhandledError" Sep 09 02:51:44.827065 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:51:44.826980 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: initializing source docker://quay.io/opendatahub/maas-api@sha256:b9449be2bc36625833e33bec983da5e2d3bac9399f129efdd3fae5bf47fab5ea: unable to retrieve auth token: invalid username/password: authentication required; artifact err: provided artifact is a container image\"" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" podUID="17f2b36e-b365-4fcc-bbe5-a617bf1b4232" Sep 09 02:51:45.079906 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:45.079797 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" event={"ID":"17f2b36e-b365-4fcc-bbe5-a617bf1b4232","Type":"ContainerStarted","Data":"94e3758da65b218d8a20b67fcf117f29f7bbc719b411f77a06cf39d2066bc5eb"} Sep 09 02:51:45.080476 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:51:45.080451 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/opendatahub/maas-api@sha256:b9449be2bc36625833e33bec983da5e2d3bac9399f129efdd3fae5bf47fab5ea\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://quay.io/opendatahub/maas-api@sha256:b9449be2bc36625833e33bec983da5e2d3bac9399f129efdd3fae5bf47fab5ea: unable to retrieve auth token: invalid username/password: authentication required; artifact err: provided artifact is a container image\"" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" podUID="17f2b36e-b365-4fcc-bbe5-a617bf1b4232" Sep 09 02:51:46.085765 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:51:46.085726 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/opendatahub/maas-api@sha256:b9449be2bc36625833e33bec983da5e2d3bac9399f129efdd3fae5bf47fab5ea\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://quay.io/opendatahub/maas-api@sha256:b9449be2bc36625833e33bec983da5e2d3bac9399f129efdd3fae5bf47fab5ea: unable to retrieve auth token: invalid username/password: authentication required; artifact err: provided artifact is a container image\"" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" podUID="17f2b36e-b365-4fcc-bbe5-a617bf1b4232" Sep 09 02:51:50.745505 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:50.745460 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:50.796552 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:50.796515 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:51:55.670736 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:51:55.670698 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:52:02.180464 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:52:02.180429 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" event={"ID":"17f2b36e-b365-4fcc-bbe5-a617bf1b4232","Type":"ContainerStarted","Data":"f91939279966ed1b2998eee631e31b984b8f9c0ae0ff60187b20fa2a4895bf40"} Sep 09 02:52:02.180901 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:52:02.180630 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" Sep 09 02:52:02.200761 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:52:02.200707 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" podStartSLOduration=1.720680172 podStartE2EDuration="19.200692136s" totalImagesPullingTime="17.480011964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-09 02:52:02.198768438 +0000 UTC m=+1310.329010537" watchObservedRunningTime="2026-09-09 02:52:02.200692136 +0000 UTC m=+1310.330934231" Sep 09 02:52:03.556205 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:52:03.556175 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:52:04.311378 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:52:04.311342 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:52:06.034450 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:52:06.034405 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:52:06.086291 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:52:06.086261 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:52:08.193160 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:52:08.193133 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" Sep 09 02:52:27.668928 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:52:27.668892 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:52:30.363197 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:52:30.363166 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:52:52.246557 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:52:52.246521 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:53:12.290166 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.290121 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb"] Sep 09 02:53:12.291460 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.291361 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" podUID="17f2b36e-b365-4fcc-bbe5-a617bf1b4232" containerName="maas-api" containerID="cri-o://f91939279966ed1b2998eee631e31b984b8f9c0ae0ff60187b20fa2a4895bf40" gracePeriod=30 Sep 09 02:53:12.314278 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.313719 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:53:12.386962 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.386925 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:53:12.445620 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.445591 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:53:12.535240 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.535202 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:53:12.560622 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.560591 2577 generic.go:356] "Generic (PLEG): container finished" podID="17f2b36e-b365-4fcc-bbe5-a617bf1b4232" containerID="f91939279966ed1b2998eee631e31b984b8f9c0ae0ff60187b20fa2a4895bf40" exitCode=0 Sep 09 02:53:12.560773 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.560663 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" event={"ID":"17f2b36e-b365-4fcc-bbe5-a617bf1b4232","Type":"ContainerDied","Data":"f91939279966ed1b2998eee631e31b984b8f9c0ae0ff60187b20fa2a4895bf40"} Sep 09 02:53:12.560773 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.560722 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" event={"ID":"17f2b36e-b365-4fcc-bbe5-a617bf1b4232","Type":"ContainerDied","Data":"94e3758da65b218d8a20b67fcf117f29f7bbc719b411f77a06cf39d2066bc5eb"} Sep 09 02:53:12.560773 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.560746 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94e3758da65b218d8a20b67fcf117f29f7bbc719b411f77a06cf39d2066bc5eb" Sep 09 02:53:12.561211 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.561193 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" Sep 09 02:53:12.605356 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.605328 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:53:12.650640 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.650612 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-maas-api-tls\") pod \"17f2b36e-b365-4fcc-bbe5-a617bf1b4232\" (UID: \"17f2b36e-b365-4fcc-bbe5-a617bf1b4232\") " Sep 09 02:53:12.650840 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.650651 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-openshift-service-ca\") pod \"17f2b36e-b365-4fcc-bbe5-a617bf1b4232\" (UID: \"17f2b36e-b365-4fcc-bbe5-a617bf1b4232\") " Sep 09 02:53:12.650840 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.650677 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-metrics-certs\") pod \"17f2b36e-b365-4fcc-bbe5-a617bf1b4232\" (UID: \"17f2b36e-b365-4fcc-bbe5-a617bf1b4232\") " Sep 09 02:53:12.650840 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.650755 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7wgj\" (UniqueName: \"kubernetes.io/projected/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-kube-api-access-r7wgj\") pod \"17f2b36e-b365-4fcc-bbe5-a617bf1b4232\" (UID: \"17f2b36e-b365-4fcc-bbe5-a617bf1b4232\") " Sep 09 02:53:12.651076 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.651046 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "17f2b36e-b365-4fcc-bbe5-a617bf1b4232" (UID: "17f2b36e-b365-4fcc-bbe5-a617bf1b4232"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:53:12.651215 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.651194 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-openshift-service-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:53:12.652995 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.652967 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "17f2b36e-b365-4fcc-bbe5-a617bf1b4232" (UID: "17f2b36e-b365-4fcc-bbe5-a617bf1b4232"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:53:12.653097 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.652976 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "17f2b36e-b365-4fcc-bbe5-a617bf1b4232" (UID: "17f2b36e-b365-4fcc-bbe5-a617bf1b4232"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:53:12.653097 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.653042 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-kube-api-access-r7wgj" (OuterVolumeSpecName: "kube-api-access-r7wgj") pod "17f2b36e-b365-4fcc-bbe5-a617bf1b4232" (UID: "17f2b36e-b365-4fcc-bbe5-a617bf1b4232"). InnerVolumeSpecName "kube-api-access-r7wgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:53:12.751921 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.751890 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-maas-api-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:53:12.751921 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.751917 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-metrics-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:53:12.751921 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:12.751927 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7wgj\" (UniqueName: \"kubernetes.io/projected/17f2b36e-b365-4fcc-bbe5-a617bf1b4232-kube-api-access-r7wgj\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:53:13.565936 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:13.565907 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb" Sep 09 02:53:13.587322 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:13.587293 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb"] Sep 09 02:53:13.591089 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:13.591068 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad1d02af-6774c6b559-r68zb"] Sep 09 02:53:14.309962 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:14.309927 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17f2b36e-b365-4fcc-bbe5-a617bf1b4232" path="/var/lib/kubelet/pods/17f2b36e-b365-4fcc-bbe5-a617bf1b4232/volumes" Sep 09 02:53:15.715697 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:15.715657 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:53:20.739982 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:20.739944 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:53:44.223156 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:44.223122 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:53:56.758347 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:56.758310 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7870-b5f33c"] Sep 09 02:53:56.758902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:56.758780 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17f2b36e-b365-4fcc-bbe5-a617bf1b4232" containerName="maas-api" Sep 09 02:53:56.758902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:56.758791 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="17f2b36e-b365-4fcc-bbe5-a617bf1b4232" containerName="maas-api" Sep 09 02:53:56.758902 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:56.758870 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="17f2b36e-b365-4fcc-bbe5-a617bf1b4232" containerName="maas-api" Sep 09 02:53:56.762050 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:56.762034 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7870-b5f33c" Sep 09 02:53:56.764593 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:56.764570 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gnkf7\"" Sep 09 02:53:56.768980 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:56.768956 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7870-b5f33c"] Sep 09 02:53:56.863572 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:56.863532 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s6bx\" (UniqueName: \"kubernetes.io/projected/1fb0be63-63cd-4e91-9399-d3a1893f5961-kube-api-access-8s6bx\") pod \"mt-curl-7870-b5f33c\" (UID: \"1fb0be63-63cd-4e91-9399-d3a1893f5961\") " pod="opendatahub/mt-curl-7870-b5f33c" Sep 09 02:53:56.964163 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:56.964129 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s6bx\" (UniqueName: \"kubernetes.io/projected/1fb0be63-63cd-4e91-9399-d3a1893f5961-kube-api-access-8s6bx\") pod \"mt-curl-7870-b5f33c\" (UID: \"1fb0be63-63cd-4e91-9399-d3a1893f5961\") " pod="opendatahub/mt-curl-7870-b5f33c" Sep 09 02:53:56.972244 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:56.972211 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s6bx\" (UniqueName: \"kubernetes.io/projected/1fb0be63-63cd-4e91-9399-d3a1893f5961-kube-api-access-8s6bx\") pod \"mt-curl-7870-b5f33c\" (UID: \"1fb0be63-63cd-4e91-9399-d3a1893f5961\") " pod="opendatahub/mt-curl-7870-b5f33c" Sep 09 02:53:57.072316 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:57.072224 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7870-b5f33c" Sep 09 02:53:57.134654 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:57.134578 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:53:57.344585 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:57.344522 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7870-b5f33c"] Sep 09 02:53:57.346945 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:53:57.346916 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fb0be63_63cd_4e91_9399_d3a1893f5961.slice/crio-147574341c8a95d1c2465e88bb389e0ed228c9bc6937cadb7dea95c74ee7d19d WatchSource:0}: Error finding container 147574341c8a95d1c2465e88bb389e0ed228c9bc6937cadb7dea95c74ee7d19d: Status 404 returned error can't find the container with id 147574341c8a95d1c2465e88bb389e0ed228c9bc6937cadb7dea95c74ee7d19d Sep 09 02:53:57.348365 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:57.348345 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 02:53:57.796752 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:57.796719 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7870-b5f33c" event={"ID":"1fb0be63-63cd-4e91-9399-d3a1893f5961","Type":"ContainerStarted","Data":"147574341c8a95d1c2465e88bb389e0ed228c9bc6937cadb7dea95c74ee7d19d"} Sep 09 02:53:57.892706 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:57.892671 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:53:58.803575 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:58.803536 2577 generic.go:356] "Generic (PLEG): container finished" podID="1fb0be63-63cd-4e91-9399-d3a1893f5961" containerID="85a00e7b1ef1db01f2c85f88035cc09c4f0191dcac66d5fb1c292da24e5321a8" exitCode=0 Sep 09 02:53:58.804052 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:58.803590 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7870-b5f33c" event={"ID":"1fb0be63-63cd-4e91-9399-d3a1893f5961","Type":"ContainerDied","Data":"85a00e7b1ef1db01f2c85f88035cc09c4f0191dcac66d5fb1c292da24e5321a8"} Sep 09 02:53:59.814799 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:59.814765 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:53:59.916499 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:59.916456 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:53:59.941508 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:59.941482 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7870-b5f33c" Sep 09 02:53:59.961904 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:59.961875 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7870-b5f33c_1fb0be63-63cd-4e91-9399-d3a1893f5961/mt-curl-7870-b5f33c/0.log" Sep 09 02:53:59.983123 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:59.983092 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7870-b5f33c"] Sep 09 02:53:59.986854 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:59.986826 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7870-b5f33c"] Sep 09 02:53:59.995120 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:59.995100 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s6bx\" (UniqueName: \"kubernetes.io/projected/1fb0be63-63cd-4e91-9399-d3a1893f5961-kube-api-access-8s6bx\") pod \"1fb0be63-63cd-4e91-9399-d3a1893f5961\" (UID: \"1fb0be63-63cd-4e91-9399-d3a1893f5961\") " Sep 09 02:53:59.997135 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:53:59.997111 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb0be63-63cd-4e91-9399-d3a1893f5961-kube-api-access-8s6bx" (OuterVolumeSpecName: "kube-api-access-8s6bx") pod "1fb0be63-63cd-4e91-9399-d3a1893f5961" (UID: "1fb0be63-63cd-4e91-9399-d3a1893f5961"). InnerVolumeSpecName "kube-api-access-8s6bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:54:00.096145 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:00.096061 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s6bx\" (UniqueName: \"kubernetes.io/projected/1fb0be63-63cd-4e91-9399-d3a1893f5961-kube-api-access-8s6bx\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:54:00.310002 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:00.309967 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fb0be63-63cd-4e91-9399-d3a1893f5961" path="/var/lib/kubelet/pods/1fb0be63-63cd-4e91-9399-d3a1893f5961/volumes" Sep 09 02:54:00.817895 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:00.817862 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7870-b5f33c" Sep 09 02:54:00.818387 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:00.817901 2577 scope.go:117] "RemoveContainer" containerID="85a00e7b1ef1db01f2c85f88035cc09c4f0191dcac66d5fb1c292da24e5321a8" Sep 09 02:54:03.373403 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:03.373372 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:07.667686 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:07.667574 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:08.746911 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:08.746878 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:09.124739 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:09.124639 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:09.184173 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:09.184146 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:09.389767 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:09.389741 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:10.636369 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:10.636333 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:10.729159 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:10.729126 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:10.942224 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:10.942145 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:11.275464 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:11.275380 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:11.446974 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:11.446938 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:11.591777 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:11.591682 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:11.783452 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:11.783420 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:13.039283 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:13.039247 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:18.863051 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:18.863015 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:18.962627 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:18.962583 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:19.568827 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:19.568783 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:25.315016 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:25.314981 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:25.421855 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:25.421804 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:25.515518 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:25.515486 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:35.105961 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:35.105931 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:35.161859 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:35.161803 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:36.040701 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:36.040672 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:36.276413 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:36.276381 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:36.374957 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:36.374852 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:54:57.242784 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:54:57.242748 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:03.099327 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:03.099285 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:03.220345 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:03.220308 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:03.452916 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:03.452886 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:03.791071 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:03.790990 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:04.233381 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:04.233349 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:07.257788 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:07.257757 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:07.357268 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:07.357231 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:08.411028 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:08.410998 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd"] Sep 09 02:55:08.411552 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:08.411535 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fb0be63-63cd-4e91-9399-d3a1893f5961" containerName="mt-curl-7870-b5f33c" Sep 09 02:55:08.411598 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:08.411555 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fb0be63-63cd-4e91-9399-d3a1893f5961" containerName="mt-curl-7870-b5f33c" Sep 09 02:55:08.411650 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:08.411641 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fb0be63-63cd-4e91-9399-d3a1893f5961" containerName="mt-curl-7870-b5f33c" Sep 09 02:55:08.414849 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:08.414832 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" Sep 09 02:55:08.417627 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:08.417609 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2b9f0d-serving-cert\"" Sep 09 02:55:08.427470 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:08.427437 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd"] Sep 09 02:55:08.446433 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:08.446399 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17568d1e-8e00-4343-bab1-22e684d2a529-metrics-certs\") pod \"maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd\" (UID: \"17568d1e-8e00-4343-bab1-22e684d2a529\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" Sep 09 02:55:08.446633 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:08.446462 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17568d1e-8e00-4343-bab1-22e684d2a529-maas-api-tls\") pod \"maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd\" (UID: \"17568d1e-8e00-4343-bab1-22e684d2a529\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" Sep 09 02:55:08.446633 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:08.446493 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:08.446633 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:08.446574 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17568d1e-8e00-4343-bab1-22e684d2a529-openshift-service-ca\") pod \"maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd\" (UID: \"17568d1e-8e00-4343-bab1-22e684d2a529\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" Sep 09 02:55:08.446633 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:08.446609 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjv4j\" (UniqueName: \"kubernetes.io/projected/17568d1e-8e00-4343-bab1-22e684d2a529-kube-api-access-tjv4j\") pod \"maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd\" (UID: \"17568d1e-8e00-4343-bab1-22e684d2a529\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" Sep 09 02:55:08.547196 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:08.547144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17568d1e-8e00-4343-bab1-22e684d2a529-metrics-certs\") pod \"maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd\" (UID: \"17568d1e-8e00-4343-bab1-22e684d2a529\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" Sep 09 02:55:08.547395 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:08.547207 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17568d1e-8e00-4343-bab1-22e684d2a529-maas-api-tls\") pod \"maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd\" (UID: \"17568d1e-8e00-4343-bab1-22e684d2a529\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" Sep 09 02:55:08.547395 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:08.547294 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17568d1e-8e00-4343-bab1-22e684d2a529-openshift-service-ca\") pod \"maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd\" (UID: \"17568d1e-8e00-4343-bab1-22e684d2a529\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" Sep 09 02:55:08.547395 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:08.547305 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:08.547579 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:08.547325 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjv4j\" (UniqueName: \"kubernetes.io/projected/17568d1e-8e00-4343-bab1-22e684d2a529-kube-api-access-tjv4j\") pod \"maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd\" (UID: \"17568d1e-8e00-4343-bab1-22e684d2a529\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" Sep 09 02:55:08.547579 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:55:08.547418 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-serving-cert: secret "maas-api-e2e-api-a-2b9f0d-serving-cert" not found Sep 09 02:55:08.547579 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:55:08.547481 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17568d1e-8e00-4343-bab1-22e684d2a529-maas-api-tls podName:17568d1e-8e00-4343-bab1-22e684d2a529 nodeName:}" failed. No retries permitted until 2026-09-09 02:55:09.047464467 +0000 UTC m=+1497.177706541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/17568d1e-8e00-4343-bab1-22e684d2a529-maas-api-tls") pod "maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" (UID: "17568d1e-8e00-4343-bab1-22e684d2a529") : secret "maas-api-e2e-api-a-2b9f0d-serving-cert" not found Sep 09 02:55:08.548196 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:08.548176 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17568d1e-8e00-4343-bab1-22e684d2a529-openshift-service-ca\") pod \"maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd\" (UID: \"17568d1e-8e00-4343-bab1-22e684d2a529\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" Sep 09 02:55:08.549841 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:08.549792 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17568d1e-8e00-4343-bab1-22e684d2a529-metrics-certs\") pod \"maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd\" (UID: \"17568d1e-8e00-4343-bab1-22e684d2a529\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" Sep 09 02:55:08.557949 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:08.557925 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjv4j\" (UniqueName: \"kubernetes.io/projected/17568d1e-8e00-4343-bab1-22e684d2a529-kube-api-access-tjv4j\") pod \"maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd\" (UID: \"17568d1e-8e00-4343-bab1-22e684d2a529\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" Sep 09 02:55:08.647197 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:08.647167 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:08.949441 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:08.949406 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:09.039649 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:09.039609 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:09.053085 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:09.053059 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17568d1e-8e00-4343-bab1-22e684d2a529-maas-api-tls\") pod \"maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd\" (UID: \"17568d1e-8e00-4343-bab1-22e684d2a529\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" Sep 09 02:55:09.055422 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:09.055400 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17568d1e-8e00-4343-bab1-22e684d2a529-maas-api-tls\") pod \"maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd\" (UID: \"17568d1e-8e00-4343-bab1-22e684d2a529\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" Sep 09 02:55:09.327129 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:09.327040 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" Sep 09 02:55:09.602005 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:09.601925 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd"] Sep 09 02:55:09.605381 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:55:09.605349 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17568d1e_8e00_4343_bab1_22e684d2a529.slice/crio-36f2ce995ab2cbcccd6e8d1c55f08d3305232b3a4922ae0f31783cc3419440cf WatchSource:0}: Error finding container 36f2ce995ab2cbcccd6e8d1c55f08d3305232b3a4922ae0f31783cc3419440cf: Status 404 returned error can't find the container with id 36f2ce995ab2cbcccd6e8d1c55f08d3305232b3a4922ae0f31783cc3419440cf Sep 09 02:55:10.199973 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:10.199935 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" event={"ID":"17568d1e-8e00-4343-bab1-22e684d2a529","Type":"ContainerStarted","Data":"2914cb8d0aa44bb3019da452dd8e28b2e9fe977ee9badc2b36e5f4b8d6bea2a3"} Sep 09 02:55:10.200159 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:10.200045 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" Sep 09 02:55:10.200159 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:10.200072 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" event={"ID":"17568d1e-8e00-4343-bab1-22e684d2a529","Type":"ContainerStarted","Data":"36f2ce995ab2cbcccd6e8d1c55f08d3305232b3a4922ae0f31783cc3419440cf"} Sep 09 02:55:10.217085 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:10.217018 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" podStartSLOduration=1.720909588 podStartE2EDuration="2.2170026s" totalImagesPullingTime="496.093012ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:55:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:55:10.216360875 +0000 UTC m=+1498.346603005" watchObservedRunningTime="2026-09-09 02:55:10.2170026 +0000 UTC m=+1498.347244709" Sep 09 02:55:12.411390 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:12.411357 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/2.log" Sep 09 02:55:12.412484 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:12.412464 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/2.log" Sep 09 02:55:16.210437 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:16.210407 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" Sep 09 02:55:19.204209 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:19.204173 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7870-395645"] Sep 09 02:55:19.207759 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:19.207738 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7870-395645" Sep 09 02:55:19.210185 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:19.210165 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gnkf7\"" Sep 09 02:55:19.214026 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:19.213999 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7870-395645"] Sep 09 02:55:19.247235 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:19.247206 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2xkd\" (UniqueName: \"kubernetes.io/projected/0ebd260d-39a4-4d3a-94a0-4f53a41004a8-kube-api-access-k2xkd\") pod \"mt-curl-7870-395645\" (UID: \"0ebd260d-39a4-4d3a-94a0-4f53a41004a8\") " pod="opendatahub/mt-curl-7870-395645" Sep 09 02:55:19.348229 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:19.348187 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2xkd\" (UniqueName: \"kubernetes.io/projected/0ebd260d-39a4-4d3a-94a0-4f53a41004a8-kube-api-access-k2xkd\") pod \"mt-curl-7870-395645\" (UID: \"0ebd260d-39a4-4d3a-94a0-4f53a41004a8\") " pod="opendatahub/mt-curl-7870-395645" Sep 09 02:55:19.358564 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:19.358529 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2xkd\" (UniqueName: \"kubernetes.io/projected/0ebd260d-39a4-4d3a-94a0-4f53a41004a8-kube-api-access-k2xkd\") pod \"mt-curl-7870-395645\" (UID: \"0ebd260d-39a4-4d3a-94a0-4f53a41004a8\") " pod="opendatahub/mt-curl-7870-395645" Sep 09 02:55:19.518883 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:19.518742 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7870-395645" Sep 09 02:55:19.789807 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:19.789783 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7870-395645"] Sep 09 02:55:19.792124 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:55:19.792095 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ebd260d_39a4_4d3a_94a0_4f53a41004a8.slice/crio-11d9d67e812cce6253b6b1872a530e8ac70a0359a517bc17d8b016575431392f WatchSource:0}: Error finding container 11d9d67e812cce6253b6b1872a530e8ac70a0359a517bc17d8b016575431392f: Status 404 returned error can't find the container with id 11d9d67e812cce6253b6b1872a530e8ac70a0359a517bc17d8b016575431392f Sep 09 02:55:20.262687 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:20.262648 2577 generic.go:356] "Generic (PLEG): container finished" podID="0ebd260d-39a4-4d3a-94a0-4f53a41004a8" containerID="a620c465e2e543f9acd8999624f57616cb87bd125fee481a6b61146690eaec52" exitCode=0 Sep 09 02:55:20.263117 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:20.262737 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7870-395645" event={"ID":"0ebd260d-39a4-4d3a-94a0-4f53a41004a8","Type":"ContainerDied","Data":"a620c465e2e543f9acd8999624f57616cb87bd125fee481a6b61146690eaec52"} Sep 09 02:55:20.263117 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:20.262787 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7870-395645" event={"ID":"0ebd260d-39a4-4d3a-94a0-4f53a41004a8","Type":"ContainerStarted","Data":"11d9d67e812cce6253b6b1872a530e8ac70a0359a517bc17d8b016575431392f"} Sep 09 02:55:21.401771 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:21.401743 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7870-395645" Sep 09 02:55:21.421952 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:21.421926 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7870-395645_0ebd260d-39a4-4d3a-94a0-4f53a41004a8/mt-curl-7870-395645/0.log" Sep 09 02:55:21.443648 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:21.443615 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7870-395645"] Sep 09 02:55:21.449521 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:21.449496 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7870-395645"] Sep 09 02:55:21.471519 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:21.471495 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2xkd\" (UniqueName: \"kubernetes.io/projected/0ebd260d-39a4-4d3a-94a0-4f53a41004a8-kube-api-access-k2xkd\") pod \"0ebd260d-39a4-4d3a-94a0-4f53a41004a8\" (UID: \"0ebd260d-39a4-4d3a-94a0-4f53a41004a8\") " Sep 09 02:55:21.473545 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:21.473517 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebd260d-39a4-4d3a-94a0-4f53a41004a8-kube-api-access-k2xkd" (OuterVolumeSpecName: "kube-api-access-k2xkd") pod "0ebd260d-39a4-4d3a-94a0-4f53a41004a8" (UID: "0ebd260d-39a4-4d3a-94a0-4f53a41004a8"). InnerVolumeSpecName "kube-api-access-k2xkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:55:21.565303 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:21.565215 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7870-cdd90c"] Sep 09 02:55:21.565889 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:21.565873 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ebd260d-39a4-4d3a-94a0-4f53a41004a8" containerName="mt-curl-7870-395645" Sep 09 02:55:21.565946 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:21.565891 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ebd260d-39a4-4d3a-94a0-4f53a41004a8" containerName="mt-curl-7870-395645" Sep 09 02:55:21.565991 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:21.565980 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ebd260d-39a4-4d3a-94a0-4f53a41004a8" containerName="mt-curl-7870-395645" Sep 09 02:55:21.569063 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:21.569048 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7870-cdd90c" Sep 09 02:55:21.572885 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:21.572859 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2xkd\" (UniqueName: \"kubernetes.io/projected/0ebd260d-39a4-4d3a-94a0-4f53a41004a8-kube-api-access-k2xkd\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:55:21.573488 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:21.573465 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7870-cdd90c"] Sep 09 02:55:21.674161 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:21.674121 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4gjd\" (UniqueName: \"kubernetes.io/projected/1a1199ab-5619-4d4c-bffe-20f15de6a785-kube-api-access-g4gjd\") pod \"mt-curl-7870-cdd90c\" (UID: \"1a1199ab-5619-4d4c-bffe-20f15de6a785\") " pod="opendatahub/mt-curl-7870-cdd90c" Sep 09 02:55:21.775617 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:21.775585 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4gjd\" (UniqueName: \"kubernetes.io/projected/1a1199ab-5619-4d4c-bffe-20f15de6a785-kube-api-access-g4gjd\") pod \"mt-curl-7870-cdd90c\" (UID: \"1a1199ab-5619-4d4c-bffe-20f15de6a785\") " pod="opendatahub/mt-curl-7870-cdd90c" Sep 09 02:55:21.783855 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:21.783830 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4gjd\" (UniqueName: \"kubernetes.io/projected/1a1199ab-5619-4d4c-bffe-20f15de6a785-kube-api-access-g4gjd\") pod \"mt-curl-7870-cdd90c\" (UID: \"1a1199ab-5619-4d4c-bffe-20f15de6a785\") " pod="opendatahub/mt-curl-7870-cdd90c" Sep 09 02:55:21.880229 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:21.880203 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7870-cdd90c" Sep 09 02:55:22.153581 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:22.153553 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7870-cdd90c"] Sep 09 02:55:22.155638 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:55:22.155615 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a1199ab_5619_4d4c_bffe_20f15de6a785.slice/crio-d97b00d4a1c09f34945e6ab117c34b8fd8b4723311463e7ed7ef27204504d91c WatchSource:0}: Error finding container d97b00d4a1c09f34945e6ab117c34b8fd8b4723311463e7ed7ef27204504d91c: Status 404 returned error can't find the container with id d97b00d4a1c09f34945e6ab117c34b8fd8b4723311463e7ed7ef27204504d91c Sep 09 02:55:22.274987 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:22.274951 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7870-cdd90c" event={"ID":"1a1199ab-5619-4d4c-bffe-20f15de6a785","Type":"ContainerStarted","Data":"d97b00d4a1c09f34945e6ab117c34b8fd8b4723311463e7ed7ef27204504d91c"} Sep 09 02:55:22.276307 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:22.276283 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7870-395645" Sep 09 02:55:22.276433 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:22.276283 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11d9d67e812cce6253b6b1872a530e8ac70a0359a517bc17d8b016575431392f" Sep 09 02:55:22.286627 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:55:22.286599 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7870-395645\" is forbidden: User \"system:node:ip-10-0-137-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-37.ec2.internal' and this object" podUID="0ebd260d-39a4-4d3a-94a0-4f53a41004a8" pod="opendatahub/mt-curl-7870-395645" Sep 09 02:55:22.309980 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:55:22.309946 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7870-395645\" is forbidden: User \"system:node:ip-10-0-137-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-37.ec2.internal' and this object" podUID="0ebd260d-39a4-4d3a-94a0-4f53a41004a8" pod="opendatahub/mt-curl-7870-395645" Sep 09 02:55:22.311741 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:22.311714 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ebd260d-39a4-4d3a-94a0-4f53a41004a8" path="/var/lib/kubelet/pods/0ebd260d-39a4-4d3a-94a0-4f53a41004a8/volumes" Sep 09 02:55:23.283381 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:23.283347 2577 generic.go:356] "Generic (PLEG): container finished" podID="1a1199ab-5619-4d4c-bffe-20f15de6a785" containerID="001f17052ff6444be68c3279750916e65afa6399ba4e85f4a6e73a93e8beb2b1" exitCode=0 Sep 09 02:55:23.283774 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:23.283436 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7870-cdd90c" event={"ID":"1a1199ab-5619-4d4c-bffe-20f15de6a785","Type":"ContainerDied","Data":"001f17052ff6444be68c3279750916e65afa6399ba4e85f4a6e73a93e8beb2b1"} Sep 09 02:55:24.423528 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:24.423503 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7870-cdd90c" Sep 09 02:55:24.442970 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:24.442945 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7870-cdd90c_1a1199ab-5619-4d4c-bffe-20f15de6a785/mt-curl-7870-cdd90c/0.log" Sep 09 02:55:24.462507 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:24.462477 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7870-cdd90c"] Sep 09 02:55:24.464549 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:24.464527 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7870-cdd90c"] Sep 09 02:55:24.501840 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:24.501789 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4gjd\" (UniqueName: \"kubernetes.io/projected/1a1199ab-5619-4d4c-bffe-20f15de6a785-kube-api-access-g4gjd\") pod \"1a1199ab-5619-4d4c-bffe-20f15de6a785\" (UID: \"1a1199ab-5619-4d4c-bffe-20f15de6a785\") " Sep 09 02:55:24.503867 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:24.503839 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1199ab-5619-4d4c-bffe-20f15de6a785-kube-api-access-g4gjd" (OuterVolumeSpecName: "kube-api-access-g4gjd") pod "1a1199ab-5619-4d4c-bffe-20f15de6a785" (UID: "1a1199ab-5619-4d4c-bffe-20f15de6a785"). InnerVolumeSpecName "kube-api-access-g4gjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:55:24.603469 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:24.603389 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4gjd\" (UniqueName: \"kubernetes.io/projected/1a1199ab-5619-4d4c-bffe-20f15de6a785-kube-api-access-g4gjd\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:55:25.295781 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:25.295752 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d97b00d4a1c09f34945e6ab117c34b8fd8b4723311463e7ed7ef27204504d91c" Sep 09 02:55:25.295781 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:25.295766 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7870-cdd90c" Sep 09 02:55:25.305960 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:55:25.305931 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7870-cdd90c\" is forbidden: User \"system:node:ip-10-0-137-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-37.ec2.internal' and this object" podUID="1a1199ab-5619-4d4c-bffe-20f15de6a785" pod="opendatahub/mt-curl-7870-cdd90c" Sep 09 02:55:26.309592 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:26.309556 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a1199ab-5619-4d4c-bffe-20f15de6a785" path="/var/lib/kubelet/pods/1a1199ab-5619-4d4c-bffe-20f15de6a785/volumes" Sep 09 02:55:27.261423 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:27.261389 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:27.548357 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:27.548276 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:32.231561 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:32.231523 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:32.737224 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:32.737176 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:32.798224 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:32.798172 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:33.061848 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:33.061751 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:33.167623 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:33.167589 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:36.546418 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:36.546324 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:36.645344 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:36.645306 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:37.692367 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:37.692328 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6"] Sep 09 02:55:37.694206 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:37.694179 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a1199ab-5619-4d4c-bffe-20f15de6a785" containerName="mt-curl-7870-cdd90c" Sep 09 02:55:37.694367 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:37.694353 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a1199ab-5619-4d4c-bffe-20f15de6a785" containerName="mt-curl-7870-cdd90c" Sep 09 02:55:37.694691 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:37.694672 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a1199ab-5619-4d4c-bffe-20f15de6a785" containerName="mt-curl-7870-cdd90c" Sep 09 02:55:37.700384 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:37.700363 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" Sep 09 02:55:37.703968 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:37.703942 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4bd452-serving-cert\"" Sep 09 02:55:37.705416 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:37.705229 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6"] Sep 09 02:55:37.722479 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:37.722450 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:37.776348 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:37.776316 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:37.836267 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:37.836236 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/282aed9a-6686-4a75-a187-bee4b4dd5588-maas-api-tls\") pod \"maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6\" (UID: \"282aed9a-6686-4a75-a187-bee4b4dd5588\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" Sep 09 02:55:37.836492 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:37.836305 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpqfz\" (UniqueName: \"kubernetes.io/projected/282aed9a-6686-4a75-a187-bee4b4dd5588-kube-api-access-qpqfz\") pod \"maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6\" (UID: \"282aed9a-6686-4a75-a187-bee4b4dd5588\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" Sep 09 02:55:37.836492 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:37.836336 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/282aed9a-6686-4a75-a187-bee4b4dd5588-metrics-certs\") pod \"maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6\" (UID: \"282aed9a-6686-4a75-a187-bee4b4dd5588\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" Sep 09 02:55:37.836492 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:37.836369 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/282aed9a-6686-4a75-a187-bee4b4dd5588-openshift-service-ca\") pod \"maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6\" (UID: \"282aed9a-6686-4a75-a187-bee4b4dd5588\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" Sep 09 02:55:37.879531 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:37.879498 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:37.937936 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:37.937900 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/282aed9a-6686-4a75-a187-bee4b4dd5588-maas-api-tls\") pod \"maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6\" (UID: \"282aed9a-6686-4a75-a187-bee4b4dd5588\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" Sep 09 02:55:37.938105 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:37.937960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpqfz\" (UniqueName: \"kubernetes.io/projected/282aed9a-6686-4a75-a187-bee4b4dd5588-kube-api-access-qpqfz\") pod \"maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6\" (UID: \"282aed9a-6686-4a75-a187-bee4b4dd5588\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" Sep 09 02:55:37.938105 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:37.937996 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/282aed9a-6686-4a75-a187-bee4b4dd5588-metrics-certs\") pod \"maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6\" (UID: \"282aed9a-6686-4a75-a187-bee4b4dd5588\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" Sep 09 02:55:37.938105 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:37.938040 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/282aed9a-6686-4a75-a187-bee4b4dd5588-openshift-service-ca\") pod \"maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6\" (UID: \"282aed9a-6686-4a75-a187-bee4b4dd5588\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" Sep 09 02:55:37.938707 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:37.938684 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/282aed9a-6686-4a75-a187-bee4b4dd5588-openshift-service-ca\") pod \"maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6\" (UID: \"282aed9a-6686-4a75-a187-bee4b4dd5588\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" Sep 09 02:55:37.940449 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:37.940427 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/282aed9a-6686-4a75-a187-bee4b4dd5588-metrics-certs\") pod \"maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6\" (UID: \"282aed9a-6686-4a75-a187-bee4b4dd5588\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" Sep 09 02:55:37.940541 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:37.940463 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/282aed9a-6686-4a75-a187-bee4b4dd5588-maas-api-tls\") pod \"maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6\" (UID: \"282aed9a-6686-4a75-a187-bee4b4dd5588\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" Sep 09 02:55:37.946171 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:37.946119 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpqfz\" (UniqueName: \"kubernetes.io/projected/282aed9a-6686-4a75-a187-bee4b4dd5588-kube-api-access-qpqfz\") pod \"maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6\" (UID: \"282aed9a-6686-4a75-a187-bee4b4dd5588\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" Sep 09 02:55:38.014190 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:38.014156 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" Sep 09 02:55:38.208873 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:38.208778 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:38.293250 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:38.293224 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6"] Sep 09 02:55:38.295738 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:55:38.295708 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod282aed9a_6686_4a75_a187_bee4b4dd5588.slice/crio-e90875b5b0dfe932cc49f39e40d2bb7c4181e7751b9ea2070793d27ed581cae9 WatchSource:0}: Error finding container e90875b5b0dfe932cc49f39e40d2bb7c4181e7751b9ea2070793d27ed581cae9: Status 404 returned error can't find the container with id e90875b5b0dfe932cc49f39e40d2bb7c4181e7751b9ea2070793d27ed581cae9 Sep 09 02:55:38.295738 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:38.295716 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:38.365753 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:38.365716 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" event={"ID":"282aed9a-6686-4a75-a187-bee4b4dd5588","Type":"ContainerStarted","Data":"e90875b5b0dfe932cc49f39e40d2bb7c4181e7751b9ea2070793d27ed581cae9"} Sep 09 02:55:39.373137 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:39.373104 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" event={"ID":"282aed9a-6686-4a75-a187-bee4b4dd5588","Type":"ContainerStarted","Data":"ef74fed440cecb6a51ebfc46b32d810bc237a4b8d01ad8835a0a0fb9541225ba"} Sep 09 02:55:39.373531 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:39.373238 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" Sep 09 02:55:39.390884 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:39.390798 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" podStartSLOduration=2.101471838 podStartE2EDuration="2.390783114s" totalImagesPullingTime="289.311276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:55:39.389141268 +0000 UTC m=+1527.519383367" watchObservedRunningTime="2026-09-09 02:55:39.390783114 +0000 UTC m=+1527.521025210" Sep 09 02:55:39.660324 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:39.660291 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:45.384285 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:45.384255 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" Sep 09 02:55:48.958574 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:48.958538 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:55.917769 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:55.917729 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:55:58.600403 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:55:58.600361 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:01.768483 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:01.768450 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:05.340130 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:05.340073 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:07.594093 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:07.594048 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:07.915969 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:07.915935 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:15.546248 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:15.546216 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:21.130128 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.130077 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd"] Sep 09 02:56:21.130574 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.130332 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" podUID="17568d1e-8e00-4343-bab1-22e684d2a529" containerName="maas-api" containerID="cri-o://2914cb8d0aa44bb3019da452dd8e28b2e9fe977ee9badc2b36e5f4b8d6bea2a3" gracePeriod=30 Sep 09 02:56:21.156539 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.156503 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:21.205925 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.205889 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" podUID="17568d1e-8e00-4343-bab1-22e684d2a529" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.70:8443/health\": dial tcp 10.133.0.70:8443: connect: connection refused" Sep 09 02:56:21.220105 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.220074 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:21.320428 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.320399 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:21.383953 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.383890 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" Sep 09 02:56:21.409363 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.409314 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:21.471640 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.471605 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17568d1e-8e00-4343-bab1-22e684d2a529-maas-api-tls\") pod \"17568d1e-8e00-4343-bab1-22e684d2a529\" (UID: \"17568d1e-8e00-4343-bab1-22e684d2a529\") " Sep 09 02:56:21.471844 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.471700 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17568d1e-8e00-4343-bab1-22e684d2a529-metrics-certs\") pod \"17568d1e-8e00-4343-bab1-22e684d2a529\" (UID: \"17568d1e-8e00-4343-bab1-22e684d2a529\") " Sep 09 02:56:21.471844 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.471750 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjv4j\" (UniqueName: \"kubernetes.io/projected/17568d1e-8e00-4343-bab1-22e684d2a529-kube-api-access-tjv4j\") pod \"17568d1e-8e00-4343-bab1-22e684d2a529\" (UID: \"17568d1e-8e00-4343-bab1-22e684d2a529\") " Sep 09 02:56:21.471844 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.471800 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17568d1e-8e00-4343-bab1-22e684d2a529-openshift-service-ca\") pod \"17568d1e-8e00-4343-bab1-22e684d2a529\" (UID: \"17568d1e-8e00-4343-bab1-22e684d2a529\") " Sep 09 02:56:21.472292 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.472248 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17568d1e-8e00-4343-bab1-22e684d2a529-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "17568d1e-8e00-4343-bab1-22e684d2a529" (UID: "17568d1e-8e00-4343-bab1-22e684d2a529"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:56:21.474045 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.474018 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17568d1e-8e00-4343-bab1-22e684d2a529-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "17568d1e-8e00-4343-bab1-22e684d2a529" (UID: "17568d1e-8e00-4343-bab1-22e684d2a529"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:56:21.474045 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.474031 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17568d1e-8e00-4343-bab1-22e684d2a529-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "17568d1e-8e00-4343-bab1-22e684d2a529" (UID: "17568d1e-8e00-4343-bab1-22e684d2a529"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:56:21.474223 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.474025 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17568d1e-8e00-4343-bab1-22e684d2a529-kube-api-access-tjv4j" (OuterVolumeSpecName: "kube-api-access-tjv4j") pod "17568d1e-8e00-4343-bab1-22e684d2a529" (UID: "17568d1e-8e00-4343-bab1-22e684d2a529"). InnerVolumeSpecName "kube-api-access-tjv4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:56:21.572645 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.572610 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17568d1e-8e00-4343-bab1-22e684d2a529-maas-api-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:56:21.572645 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.572637 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17568d1e-8e00-4343-bab1-22e684d2a529-metrics-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:56:21.572645 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.572650 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjv4j\" (UniqueName: \"kubernetes.io/projected/17568d1e-8e00-4343-bab1-22e684d2a529-kube-api-access-tjv4j\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:56:21.572911 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.572663 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17568d1e-8e00-4343-bab1-22e684d2a529-openshift-service-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:56:21.610866 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.610809 2577 generic.go:356] "Generic (PLEG): container finished" podID="17568d1e-8e00-4343-bab1-22e684d2a529" containerID="2914cb8d0aa44bb3019da452dd8e28b2e9fe977ee9badc2b36e5f4b8d6bea2a3" exitCode=0 Sep 09 02:56:21.611027 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.610876 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" Sep 09 02:56:21.611027 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.610908 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" event={"ID":"17568d1e-8e00-4343-bab1-22e684d2a529","Type":"ContainerDied","Data":"2914cb8d0aa44bb3019da452dd8e28b2e9fe977ee9badc2b36e5f4b8d6bea2a3"} Sep 09 02:56:21.611027 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.610966 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd" event={"ID":"17568d1e-8e00-4343-bab1-22e684d2a529","Type":"ContainerDied","Data":"36f2ce995ab2cbcccd6e8d1c55f08d3305232b3a4922ae0f31783cc3419440cf"} Sep 09 02:56:21.611027 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.610989 2577 scope.go:117] "RemoveContainer" containerID="2914cb8d0aa44bb3019da452dd8e28b2e9fe977ee9badc2b36e5f4b8d6bea2a3" Sep 09 02:56:21.620575 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.620557 2577 scope.go:117] "RemoveContainer" containerID="2914cb8d0aa44bb3019da452dd8e28b2e9fe977ee9badc2b36e5f4b8d6bea2a3" Sep 09 02:56:21.620883 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:56:21.620863 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2914cb8d0aa44bb3019da452dd8e28b2e9fe977ee9badc2b36e5f4b8d6bea2a3\": container with ID starting with 2914cb8d0aa44bb3019da452dd8e28b2e9fe977ee9badc2b36e5f4b8d6bea2a3 not found: ID does not exist" containerID="2914cb8d0aa44bb3019da452dd8e28b2e9fe977ee9badc2b36e5f4b8d6bea2a3" Sep 09 02:56:21.620968 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.620897 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2914cb8d0aa44bb3019da452dd8e28b2e9fe977ee9badc2b36e5f4b8d6bea2a3"} err="failed to get container status \"2914cb8d0aa44bb3019da452dd8e28b2e9fe977ee9badc2b36e5f4b8d6bea2a3\": rpc error: code = NotFound desc = could not find container \"2914cb8d0aa44bb3019da452dd8e28b2e9fe977ee9badc2b36e5f4b8d6bea2a3\": container with ID starting with 2914cb8d0aa44bb3019da452dd8e28b2e9fe977ee9badc2b36e5f4b8d6bea2a3 not found: ID does not exist" Sep 09 02:56:21.634402 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.634337 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd"] Sep 09 02:56:21.637540 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.637515 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2b9f0d-54894ff69d-5tlfd"] Sep 09 02:56:21.759617 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:21.759583 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:22.311349 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:22.311312 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17568d1e-8e00-4343-bab1-22e684d2a529" path="/var/lib/kubelet/pods/17568d1e-8e00-4343-bab1-22e684d2a529/volumes" Sep 09 02:56:23.815426 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:23.815390 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:27.513980 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:27.513930 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:27.613983 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:27.613944 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:27.799273 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:27.799191 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:28.436307 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:28.436270 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:38.981561 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:38.981510 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:39.871173 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:39.871138 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:40.339736 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:40.339703 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:40.673970 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:40.673936 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:40.974884 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:40.974791 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:43.812408 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:43.812374 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6"] Sep 09 02:56:43.812806 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:43.812662 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" podUID="282aed9a-6686-4a75-a187-bee4b4dd5588" containerName="maas-api" containerID="cri-o://ef74fed440cecb6a51ebfc46b32d810bc237a4b8d01ad8835a0a0fb9541225ba" gracePeriod=30 Sep 09 02:56:43.840473 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:43.840426 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:43.912061 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:43.912027 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:44.012387 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.012343 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:44.054124 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.054103 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" Sep 09 02:56:44.079985 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.079907 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:44.147175 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.147146 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:44.190359 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.190334 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/282aed9a-6686-4a75-a187-bee4b4dd5588-maas-api-tls\") pod \"282aed9a-6686-4a75-a187-bee4b4dd5588\" (UID: \"282aed9a-6686-4a75-a187-bee4b4dd5588\") " Sep 09 02:56:44.190537 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.190380 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/282aed9a-6686-4a75-a187-bee4b4dd5588-metrics-certs\") pod \"282aed9a-6686-4a75-a187-bee4b4dd5588\" (UID: \"282aed9a-6686-4a75-a187-bee4b4dd5588\") " Sep 09 02:56:44.190537 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.190441 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpqfz\" (UniqueName: \"kubernetes.io/projected/282aed9a-6686-4a75-a187-bee4b4dd5588-kube-api-access-qpqfz\") pod \"282aed9a-6686-4a75-a187-bee4b4dd5588\" (UID: \"282aed9a-6686-4a75-a187-bee4b4dd5588\") " Sep 09 02:56:44.190537 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.190475 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/282aed9a-6686-4a75-a187-bee4b4dd5588-openshift-service-ca\") pod \"282aed9a-6686-4a75-a187-bee4b4dd5588\" (UID: \"282aed9a-6686-4a75-a187-bee4b4dd5588\") " Sep 09 02:56:44.190901 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.190872 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/282aed9a-6686-4a75-a187-bee4b4dd5588-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "282aed9a-6686-4a75-a187-bee4b4dd5588" (UID: "282aed9a-6686-4a75-a187-bee4b4dd5588"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:56:44.192463 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.192439 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/282aed9a-6686-4a75-a187-bee4b4dd5588-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "282aed9a-6686-4a75-a187-bee4b4dd5588" (UID: "282aed9a-6686-4a75-a187-bee4b4dd5588"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:56:44.192572 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.192550 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/282aed9a-6686-4a75-a187-bee4b4dd5588-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "282aed9a-6686-4a75-a187-bee4b4dd5588" (UID: "282aed9a-6686-4a75-a187-bee4b4dd5588"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:56:44.192618 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.192585 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/282aed9a-6686-4a75-a187-bee4b4dd5588-kube-api-access-qpqfz" (OuterVolumeSpecName: "kube-api-access-qpqfz") pod "282aed9a-6686-4a75-a187-bee4b4dd5588" (UID: "282aed9a-6686-4a75-a187-bee4b4dd5588"). InnerVolumeSpecName "kube-api-access-qpqfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:56:44.292318 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.292286 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/282aed9a-6686-4a75-a187-bee4b4dd5588-maas-api-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:56:44.292318 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.292313 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/282aed9a-6686-4a75-a187-bee4b4dd5588-metrics-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:56:44.292318 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.292322 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpqfz\" (UniqueName: \"kubernetes.io/projected/282aed9a-6686-4a75-a187-bee4b4dd5588-kube-api-access-qpqfz\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:56:44.292579 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.292332 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/282aed9a-6686-4a75-a187-bee4b4dd5588-openshift-service-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:56:44.486456 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.486420 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:44.748043 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.747952 2577 generic.go:356] "Generic (PLEG): container finished" podID="282aed9a-6686-4a75-a187-bee4b4dd5588" containerID="ef74fed440cecb6a51ebfc46b32d810bc237a4b8d01ad8835a0a0fb9541225ba" exitCode=0 Sep 09 02:56:44.748043 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.748003 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" Sep 09 02:56:44.748276 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.748042 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" event={"ID":"282aed9a-6686-4a75-a187-bee4b4dd5588","Type":"ContainerDied","Data":"ef74fed440cecb6a51ebfc46b32d810bc237a4b8d01ad8835a0a0fb9541225ba"} Sep 09 02:56:44.748276 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.748103 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6" event={"ID":"282aed9a-6686-4a75-a187-bee4b4dd5588","Type":"ContainerDied","Data":"e90875b5b0dfe932cc49f39e40d2bb7c4181e7751b9ea2070793d27ed581cae9"} Sep 09 02:56:44.748276 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.748131 2577 scope.go:117] "RemoveContainer" containerID="ef74fed440cecb6a51ebfc46b32d810bc237a4b8d01ad8835a0a0fb9541225ba" Sep 09 02:56:44.760233 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.760211 2577 scope.go:117] "RemoveContainer" containerID="ef74fed440cecb6a51ebfc46b32d810bc237a4b8d01ad8835a0a0fb9541225ba" Sep 09 02:56:44.760508 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:56:44.760490 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef74fed440cecb6a51ebfc46b32d810bc237a4b8d01ad8835a0a0fb9541225ba\": container with ID starting with ef74fed440cecb6a51ebfc46b32d810bc237a4b8d01ad8835a0a0fb9541225ba not found: ID does not exist" containerID="ef74fed440cecb6a51ebfc46b32d810bc237a4b8d01ad8835a0a0fb9541225ba" Sep 09 02:56:44.760554 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.760515 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef74fed440cecb6a51ebfc46b32d810bc237a4b8d01ad8835a0a0fb9541225ba"} err="failed to get container status \"ef74fed440cecb6a51ebfc46b32d810bc237a4b8d01ad8835a0a0fb9541225ba\": rpc error: code = NotFound desc = could not find container \"ef74fed440cecb6a51ebfc46b32d810bc237a4b8d01ad8835a0a0fb9541225ba\": container with ID starting with ef74fed440cecb6a51ebfc46b32d810bc237a4b8d01ad8835a0a0fb9541225ba not found: ID does not exist" Sep 09 02:56:44.764744 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.764720 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6"] Sep 09 02:56:44.768371 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:44.768346 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4bd452-5b5fbfd877-9ksn6"] Sep 09 02:56:45.096002 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.095887 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh"] Sep 09 02:56:45.096455 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.096332 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17568d1e-8e00-4343-bab1-22e684d2a529" containerName="maas-api" Sep 09 02:56:45.096455 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.096344 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="17568d1e-8e00-4343-bab1-22e684d2a529" containerName="maas-api" Sep 09 02:56:45.096455 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.096360 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="282aed9a-6686-4a75-a187-bee4b4dd5588" containerName="maas-api" Sep 09 02:56:45.096455 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.096366 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="282aed9a-6686-4a75-a187-bee4b4dd5588" containerName="maas-api" Sep 09 02:56:45.096714 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.096472 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="282aed9a-6686-4a75-a187-bee4b4dd5588" containerName="maas-api" Sep 09 02:56:45.096714 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.096499 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="17568d1e-8e00-4343-bab1-22e684d2a529" containerName="maas-api" Sep 09 02:56:45.101347 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.101324 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" Sep 09 02:56:45.103768 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.103748 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2085ee-serving-cert\"" Sep 09 02:56:45.113289 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.113265 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh"] Sep 09 02:56:45.138777 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.138750 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:45.200752 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.200716 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2085ee-5484646457-brxgh\" (UID: \"38c3ad74-d112-4a2f-a7e1-8fc14b02853b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" Sep 09 02:56:45.200928 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.200765 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2085ee-5484646457-brxgh\" (UID: \"38c3ad74-d112-4a2f-a7e1-8fc14b02853b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" Sep 09 02:56:45.200928 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.200847 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-metrics-certs\") pod \"maas-api-e2e-ipp-a-2085ee-5484646457-brxgh\" (UID: \"38c3ad74-d112-4a2f-a7e1-8fc14b02853b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" Sep 09 02:56:45.200928 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.200901 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl5ts\" (UniqueName: \"kubernetes.io/projected/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-kube-api-access-xl5ts\") pod \"maas-api-e2e-ipp-a-2085ee-5484646457-brxgh\" (UID: \"38c3ad74-d112-4a2f-a7e1-8fc14b02853b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" Sep 09 02:56:45.235260 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.235228 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:45.301640 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.301610 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2085ee-5484646457-brxgh\" (UID: \"38c3ad74-d112-4a2f-a7e1-8fc14b02853b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" Sep 09 02:56:45.301874 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.301662 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2085ee-5484646457-brxgh\" (UID: \"38c3ad74-d112-4a2f-a7e1-8fc14b02853b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" Sep 09 02:56:45.301874 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.301690 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-metrics-certs\") pod \"maas-api-e2e-ipp-a-2085ee-5484646457-brxgh\" (UID: \"38c3ad74-d112-4a2f-a7e1-8fc14b02853b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" Sep 09 02:56:45.301874 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.301718 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl5ts\" (UniqueName: \"kubernetes.io/projected/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-kube-api-access-xl5ts\") pod \"maas-api-e2e-ipp-a-2085ee-5484646457-brxgh\" (UID: \"38c3ad74-d112-4a2f-a7e1-8fc14b02853b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" Sep 09 02:56:45.301874 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:56:45.301778 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-serving-cert: secret "maas-api-e2e-ipp-a-2085ee-serving-cert" not found Sep 09 02:56:45.302099 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:56:45.301889 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-maas-api-tls podName:38c3ad74-d112-4a2f-a7e1-8fc14b02853b nodeName:}" failed. No retries permitted until 2026-09-09 02:56:45.801862882 +0000 UTC m=+1593.932105003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-maas-api-tls") pod "maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" (UID: "38c3ad74-d112-4a2f-a7e1-8fc14b02853b") : secret "maas-api-e2e-ipp-a-2085ee-serving-cert" not found Sep 09 02:56:45.302356 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.302335 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2085ee-5484646457-brxgh\" (UID: \"38c3ad74-d112-4a2f-a7e1-8fc14b02853b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" Sep 09 02:56:45.304157 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.304139 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-metrics-certs\") pod \"maas-api-e2e-ipp-a-2085ee-5484646457-brxgh\" (UID: \"38c3ad74-d112-4a2f-a7e1-8fc14b02853b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" Sep 09 02:56:45.312284 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.312259 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl5ts\" (UniqueName: \"kubernetes.io/projected/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-kube-api-access-xl5ts\") pod \"maas-api-e2e-ipp-a-2085ee-5484646457-brxgh\" (UID: \"38c3ad74-d112-4a2f-a7e1-8fc14b02853b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" Sep 09 02:56:45.509026 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.508990 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:45.722722 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.722689 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:45.807870 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.807778 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2085ee-5484646457-brxgh\" (UID: \"38c3ad74-d112-4a2f-a7e1-8fc14b02853b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" Sep 09 02:56:45.810114 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.810095 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2085ee-5484646457-brxgh\" (UID: \"38c3ad74-d112-4a2f-a7e1-8fc14b02853b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" Sep 09 02:56:45.823334 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:45.823310 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:46.015543 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:46.015509 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" Sep 09 02:56:46.291335 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:46.291240 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh"] Sep 09 02:56:46.293793 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:56:46.293768 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c3ad74_d112_4a2f_a7e1_8fc14b02853b.slice/crio-de5a218ddcece4d1acb4f625218dc659bd736ce9f3331dc88e5e45cdab6e7ee1 WatchSource:0}: Error finding container de5a218ddcece4d1acb4f625218dc659bd736ce9f3331dc88e5e45cdab6e7ee1: Status 404 returned error can't find the container with id de5a218ddcece4d1acb4f625218dc659bd736ce9f3331dc88e5e45cdab6e7ee1 Sep 09 02:56:46.310875 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:46.310848 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="282aed9a-6686-4a75-a187-bee4b4dd5588" path="/var/lib/kubelet/pods/282aed9a-6686-4a75-a187-bee4b4dd5588/volumes" Sep 09 02:56:46.763890 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:46.763848 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" event={"ID":"38c3ad74-d112-4a2f-a7e1-8fc14b02853b","Type":"ContainerStarted","Data":"af97c7fcd2104d4d05a5b6c75633fc0716e144a4cba006c631deacb579873faf"} Sep 09 02:56:46.764089 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:46.763906 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" event={"ID":"38c3ad74-d112-4a2f-a7e1-8fc14b02853b","Type":"ContainerStarted","Data":"de5a218ddcece4d1acb4f625218dc659bd736ce9f3331dc88e5e45cdab6e7ee1"} Sep 09 02:56:46.764089 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:46.764039 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" Sep 09 02:56:46.784744 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:46.784698 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" podStartSLOduration=1.406188213 podStartE2EDuration="1.784681946s" totalImagesPullingTime="378.493733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:56:46.781973417 +0000 UTC m=+1594.912215516" watchObservedRunningTime="2026-09-09 02:56:46.784681946 +0000 UTC m=+1594.914924041" Sep 09 02:56:50.258544 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:50.258506 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:50.358756 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:50.358723 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:50.539181 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:50.538997 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:51.053308 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:51.053274 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:56:52.775778 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:56:52.775751 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" Sep 09 02:57:01.368580 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.368545 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9"] Sep 09 02:57:01.369092 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.368790 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" podUID="623fc653-8a55-4a7a-87c9-b5e949e147aa" containerName="maas-api" containerID="cri-o://93aa8674cf5fc4b1bc03f34776e50580f6a398e32011b0a99e1c9eeb35aab928" gracePeriod=30 Sep 09 02:57:01.401521 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.401472 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:01.463377 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.463340 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:01.530070 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.530023 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:01.592110 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.592079 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:01.612493 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.612474 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" Sep 09 02:57:01.654155 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.654122 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:01.768800 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.768762 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/623fc653-8a55-4a7a-87c9-b5e949e147aa-openshift-service-ca\") pod \"623fc653-8a55-4a7a-87c9-b5e949e147aa\" (UID: \"623fc653-8a55-4a7a-87c9-b5e949e147aa\") " Sep 09 02:57:01.769005 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.768883 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8qsq\" (UniqueName: \"kubernetes.io/projected/623fc653-8a55-4a7a-87c9-b5e949e147aa-kube-api-access-w8qsq\") pod \"623fc653-8a55-4a7a-87c9-b5e949e147aa\" (UID: \"623fc653-8a55-4a7a-87c9-b5e949e147aa\") " Sep 09 02:57:01.769005 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.768914 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623fc653-8a55-4a7a-87c9-b5e949e147aa-metrics-certs\") pod \"623fc653-8a55-4a7a-87c9-b5e949e147aa\" (UID: \"623fc653-8a55-4a7a-87c9-b5e949e147aa\") " Sep 09 02:57:01.769005 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.768945 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/623fc653-8a55-4a7a-87c9-b5e949e147aa-maas-api-tls\") pod \"623fc653-8a55-4a7a-87c9-b5e949e147aa\" (UID: \"623fc653-8a55-4a7a-87c9-b5e949e147aa\") " Sep 09 02:57:01.769192 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.769126 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/623fc653-8a55-4a7a-87c9-b5e949e147aa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "623fc653-8a55-4a7a-87c9-b5e949e147aa" (UID: "623fc653-8a55-4a7a-87c9-b5e949e147aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:57:01.769269 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.769248 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/623fc653-8a55-4a7a-87c9-b5e949e147aa-openshift-service-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:57:01.771058 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.771028 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/623fc653-8a55-4a7a-87c9-b5e949e147aa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "623fc653-8a55-4a7a-87c9-b5e949e147aa" (UID: "623fc653-8a55-4a7a-87c9-b5e949e147aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:57:01.771058 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.771050 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/623fc653-8a55-4a7a-87c9-b5e949e147aa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "623fc653-8a55-4a7a-87c9-b5e949e147aa" (UID: "623fc653-8a55-4a7a-87c9-b5e949e147aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:57:01.771212 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.771052 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623fc653-8a55-4a7a-87c9-b5e949e147aa-kube-api-access-w8qsq" (OuterVolumeSpecName: "kube-api-access-w8qsq") pod "623fc653-8a55-4a7a-87c9-b5e949e147aa" (UID: "623fc653-8a55-4a7a-87c9-b5e949e147aa"). InnerVolumeSpecName "kube-api-access-w8qsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:57:01.851537 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.851498 2577 generic.go:356] "Generic (PLEG): container finished" podID="623fc653-8a55-4a7a-87c9-b5e949e147aa" containerID="93aa8674cf5fc4b1bc03f34776e50580f6a398e32011b0a99e1c9eeb35aab928" exitCode=0 Sep 09 02:57:01.851687 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.851551 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" Sep 09 02:57:01.851687 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.851565 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" event={"ID":"623fc653-8a55-4a7a-87c9-b5e949e147aa","Type":"ContainerDied","Data":"93aa8674cf5fc4b1bc03f34776e50580f6a398e32011b0a99e1c9eeb35aab928"} Sep 09 02:57:01.851687 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.851621 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9" event={"ID":"623fc653-8a55-4a7a-87c9-b5e949e147aa","Type":"ContainerDied","Data":"8907c00c452b16ea68896c00cba1bba3a57832abe3bf6603f283663b7fe30380"} Sep 09 02:57:01.851687 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.851653 2577 scope.go:117] "RemoveContainer" containerID="93aa8674cf5fc4b1bc03f34776e50580f6a398e32011b0a99e1c9eeb35aab928" Sep 09 02:57:01.861615 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.861598 2577 scope.go:117] "RemoveContainer" containerID="93aa8674cf5fc4b1bc03f34776e50580f6a398e32011b0a99e1c9eeb35aab928" Sep 09 02:57:01.861910 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:57:01.861890 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93aa8674cf5fc4b1bc03f34776e50580f6a398e32011b0a99e1c9eeb35aab928\": container with ID starting with 93aa8674cf5fc4b1bc03f34776e50580f6a398e32011b0a99e1c9eeb35aab928 not found: ID does not exist" containerID="93aa8674cf5fc4b1bc03f34776e50580f6a398e32011b0a99e1c9eeb35aab928" Sep 09 02:57:01.861965 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.861919 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93aa8674cf5fc4b1bc03f34776e50580f6a398e32011b0a99e1c9eeb35aab928"} err="failed to get container status \"93aa8674cf5fc4b1bc03f34776e50580f6a398e32011b0a99e1c9eeb35aab928\": rpc error: code = NotFound desc = could not find container \"93aa8674cf5fc4b1bc03f34776e50580f6a398e32011b0a99e1c9eeb35aab928\": container with ID starting with 93aa8674cf5fc4b1bc03f34776e50580f6a398e32011b0a99e1c9eeb35aab928 not found: ID does not exist" Sep 09 02:57:01.870428 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.870405 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623fc653-8a55-4a7a-87c9-b5e949e147aa-metrics-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:57:01.870428 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.870430 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/623fc653-8a55-4a7a-87c9-b5e949e147aa-maas-api-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:57:01.870628 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.870444 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8qsq\" (UniqueName: \"kubernetes.io/projected/623fc653-8a55-4a7a-87c9-b5e949e147aa-kube-api-access-w8qsq\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:57:01.873312 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.873290 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9"] Sep 09 02:57:01.877121 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:01.877103 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d838aa-677f9b8b45-kzvh9"] Sep 09 02:57:02.018178 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:02.018093 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:02.310740 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:02.310655 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="623fc653-8a55-4a7a-87c9-b5e949e147aa" path="/var/lib/kubelet/pods/623fc653-8a55-4a7a-87c9-b5e949e147aa/volumes" Sep 09 02:57:04.311138 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:04.311102 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:07.165938 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:07.165849 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:08.007343 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:08.007314 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:08.108488 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:08.108458 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:08.311707 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:08.311621 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:09.045856 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:09.045798 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:09.146417 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:09.146387 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:09.807493 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:09.807455 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:10.167250 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:10.167212 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:10.314944 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:10.314907 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:13.172691 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:13.172654 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:13.264903 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:13.264872 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:14.407730 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.407700 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc"] Sep 09 02:57:14.408423 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.408400 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="623fc653-8a55-4a7a-87c9-b5e949e147aa" containerName="maas-api" Sep 09 02:57:14.408423 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.408422 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="623fc653-8a55-4a7a-87c9-b5e949e147aa" containerName="maas-api" Sep 09 02:57:14.408620 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.408546 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="623fc653-8a55-4a7a-87c9-b5e949e147aa" containerName="maas-api" Sep 09 02:57:14.411698 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.411678 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" Sep 09 02:57:14.414183 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.414156 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-621ac9-serving-cert\"" Sep 09 02:57:14.422488 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.422462 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc"] Sep 09 02:57:14.444728 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.444694 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:14.476912 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.476881 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18778870-20d2-4e85-b4c8-52dbd1933b24-maas-api-tls\") pod \"maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc\" (UID: \"18778870-20d2-4e85-b4c8-52dbd1933b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" Sep 09 02:57:14.477063 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.476936 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-855vf\" (UniqueName: \"kubernetes.io/projected/18778870-20d2-4e85-b4c8-52dbd1933b24-kube-api-access-855vf\") pod \"maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc\" (UID: \"18778870-20d2-4e85-b4c8-52dbd1933b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" Sep 09 02:57:14.477063 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.477021 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18778870-20d2-4e85-b4c8-52dbd1933b24-metrics-certs\") pod \"maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc\" (UID: \"18778870-20d2-4e85-b4c8-52dbd1933b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" Sep 09 02:57:14.477156 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.477091 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18778870-20d2-4e85-b4c8-52dbd1933b24-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc\" (UID: \"18778870-20d2-4e85-b4c8-52dbd1933b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" Sep 09 02:57:14.545554 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.545525 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:14.578191 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.578161 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18778870-20d2-4e85-b4c8-52dbd1933b24-maas-api-tls\") pod \"maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc\" (UID: \"18778870-20d2-4e85-b4c8-52dbd1933b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" Sep 09 02:57:14.578349 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.578201 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-855vf\" (UniqueName: \"kubernetes.io/projected/18778870-20d2-4e85-b4c8-52dbd1933b24-kube-api-access-855vf\") pod \"maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc\" (UID: \"18778870-20d2-4e85-b4c8-52dbd1933b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" Sep 09 02:57:14.578349 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.578308 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18778870-20d2-4e85-b4c8-52dbd1933b24-metrics-certs\") pod \"maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc\" (UID: \"18778870-20d2-4e85-b4c8-52dbd1933b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" Sep 09 02:57:14.578446 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.578374 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18778870-20d2-4e85-b4c8-52dbd1933b24-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc\" (UID: \"18778870-20d2-4e85-b4c8-52dbd1933b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" Sep 09 02:57:14.579048 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.579025 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18778870-20d2-4e85-b4c8-52dbd1933b24-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc\" (UID: \"18778870-20d2-4e85-b4c8-52dbd1933b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" Sep 09 02:57:14.580647 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.580627 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18778870-20d2-4e85-b4c8-52dbd1933b24-metrics-certs\") pod \"maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc\" (UID: \"18778870-20d2-4e85-b4c8-52dbd1933b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" Sep 09 02:57:14.580742 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.580680 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18778870-20d2-4e85-b4c8-52dbd1933b24-maas-api-tls\") pod \"maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc\" (UID: \"18778870-20d2-4e85-b4c8-52dbd1933b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" Sep 09 02:57:14.586154 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.586129 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-855vf\" (UniqueName: \"kubernetes.io/projected/18778870-20d2-4e85-b4c8-52dbd1933b24-kube-api-access-855vf\") pod \"maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc\" (UID: \"18778870-20d2-4e85-b4c8-52dbd1933b24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" Sep 09 02:57:14.644829 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.644791 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:14.725119 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.725052 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" Sep 09 02:57:14.914552 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:14.914521 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:15.011417 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:15.011327 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc"] Sep 09 02:57:15.016413 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:15.016363 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:15.020363 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:57:15.020339 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18778870_20d2_4e85_b4c8_52dbd1933b24.slice/crio-46a73e5ef4db9985a5613d2094083585e4f24b9e47788852c85fcfaf0dee1ac7 WatchSource:0}: Error finding container 46a73e5ef4db9985a5613d2094083585e4f24b9e47788852c85fcfaf0dee1ac7: Status 404 returned error can't find the container with id 46a73e5ef4db9985a5613d2094083585e4f24b9e47788852c85fcfaf0dee1ac7 Sep 09 02:57:15.930679 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:15.930646 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" event={"ID":"18778870-20d2-4e85-b4c8-52dbd1933b24","Type":"ContainerStarted","Data":"f4601e07b45a14fd2430fe0f1621321a6530dc0b8b22a28e0ddabbd1d1ae2f61"} Sep 09 02:57:15.931067 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:15.930697 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" event={"ID":"18778870-20d2-4e85-b4c8-52dbd1933b24","Type":"ContainerStarted","Data":"46a73e5ef4db9985a5613d2094083585e4f24b9e47788852c85fcfaf0dee1ac7"} Sep 09 02:57:15.931067 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:15.930743 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" Sep 09 02:57:15.950179 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:15.950134 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" podStartSLOduration=1.5993502 podStartE2EDuration="1.950120228s" totalImagesPullingTime="350.770028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:57:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:57:15.947966795 +0000 UTC m=+1624.078208892" watchObservedRunningTime="2026-09-09 02:57:15.950120228 +0000 UTC m=+1624.080362327" Sep 09 02:57:18.822262 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:18.822223 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:19.399687 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.399655 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk"] Sep 09 02:57:19.399949 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.399923 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" podUID="655f79a2-a584-44f2-a615-2cad39a3a608" containerName="maas-api" containerID="cri-o://858bee7071c9dcb3a71128974131fbb4b2404cfe479488dc872152ae98ac49b0" gracePeriod=30 Sep 09 02:57:19.428742 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.428696 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:19.512945 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.512810 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:19.572435 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.572364 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:19.635023 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.634996 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:19.646114 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.646094 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" Sep 09 02:57:19.699384 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.699310 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:19.729617 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.729590 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655f79a2-a584-44f2-a615-2cad39a3a608-metrics-certs\") pod \"655f79a2-a584-44f2-a615-2cad39a3a608\" (UID: \"655f79a2-a584-44f2-a615-2cad39a3a608\") " Sep 09 02:57:19.729761 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.729642 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655f79a2-a584-44f2-a615-2cad39a3a608-maas-api-tls\") pod \"655f79a2-a584-44f2-a615-2cad39a3a608\" (UID: \"655f79a2-a584-44f2-a615-2cad39a3a608\") " Sep 09 02:57:19.729761 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.729670 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655f79a2-a584-44f2-a615-2cad39a3a608-openshift-service-ca\") pod \"655f79a2-a584-44f2-a615-2cad39a3a608\" (UID: \"655f79a2-a584-44f2-a615-2cad39a3a608\") " Sep 09 02:57:19.729761 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.729723 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkwgd\" (UniqueName: \"kubernetes.io/projected/655f79a2-a584-44f2-a615-2cad39a3a608-kube-api-access-lkwgd\") pod \"655f79a2-a584-44f2-a615-2cad39a3a608\" (UID: \"655f79a2-a584-44f2-a615-2cad39a3a608\") " Sep 09 02:57:19.730117 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.730091 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655f79a2-a584-44f2-a615-2cad39a3a608-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "655f79a2-a584-44f2-a615-2cad39a3a608" (UID: "655f79a2-a584-44f2-a615-2cad39a3a608"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:57:19.731854 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.731831 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/655f79a2-a584-44f2-a615-2cad39a3a608-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "655f79a2-a584-44f2-a615-2cad39a3a608" (UID: "655f79a2-a584-44f2-a615-2cad39a3a608"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:57:19.731940 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.731897 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/655f79a2-a584-44f2-a615-2cad39a3a608-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "655f79a2-a584-44f2-a615-2cad39a3a608" (UID: "655f79a2-a584-44f2-a615-2cad39a3a608"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:57:19.731940 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.731901 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655f79a2-a584-44f2-a615-2cad39a3a608-kube-api-access-lkwgd" (OuterVolumeSpecName: "kube-api-access-lkwgd") pod "655f79a2-a584-44f2-a615-2cad39a3a608" (UID: "655f79a2-a584-44f2-a615-2cad39a3a608"). InnerVolumeSpecName "kube-api-access-lkwgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:57:19.831280 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.831250 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655f79a2-a584-44f2-a615-2cad39a3a608-maas-api-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:57:19.831280 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.831279 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655f79a2-a584-44f2-a615-2cad39a3a608-openshift-service-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:57:19.831692 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.831289 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkwgd\" (UniqueName: \"kubernetes.io/projected/655f79a2-a584-44f2-a615-2cad39a3a608-kube-api-access-lkwgd\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:57:19.831692 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.831299 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655f79a2-a584-44f2-a615-2cad39a3a608-metrics-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:57:19.952246 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.952150 2577 generic.go:356] "Generic (PLEG): container finished" podID="655f79a2-a584-44f2-a615-2cad39a3a608" containerID="858bee7071c9dcb3a71128974131fbb4b2404cfe479488dc872152ae98ac49b0" exitCode=0 Sep 09 02:57:19.952246 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.952194 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" Sep 09 02:57:19.952246 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.952230 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" event={"ID":"655f79a2-a584-44f2-a615-2cad39a3a608","Type":"ContainerDied","Data":"858bee7071c9dcb3a71128974131fbb4b2404cfe479488dc872152ae98ac49b0"} Sep 09 02:57:19.952505 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.952272 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk" event={"ID":"655f79a2-a584-44f2-a615-2cad39a3a608","Type":"ContainerDied","Data":"0d27e3c57212f67bca1bad3a9f23637774e51844ac962d9245dbdfd4997ca6dc"} Sep 09 02:57:19.952505 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.952294 2577 scope.go:117] "RemoveContainer" containerID="858bee7071c9dcb3a71128974131fbb4b2404cfe479488dc872152ae98ac49b0" Sep 09 02:57:19.963899 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.963876 2577 scope.go:117] "RemoveContainer" containerID="858bee7071c9dcb3a71128974131fbb4b2404cfe479488dc872152ae98ac49b0" Sep 09 02:57:19.964181 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:57:19.964159 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"858bee7071c9dcb3a71128974131fbb4b2404cfe479488dc872152ae98ac49b0\": container with ID starting with 858bee7071c9dcb3a71128974131fbb4b2404cfe479488dc872152ae98ac49b0 not found: ID does not exist" containerID="858bee7071c9dcb3a71128974131fbb4b2404cfe479488dc872152ae98ac49b0" Sep 09 02:57:19.964268 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.964188 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858bee7071c9dcb3a71128974131fbb4b2404cfe479488dc872152ae98ac49b0"} err="failed to get container status \"858bee7071c9dcb3a71128974131fbb4b2404cfe479488dc872152ae98ac49b0\": rpc error: code = NotFound desc = could not find container \"858bee7071c9dcb3a71128974131fbb4b2404cfe479488dc872152ae98ac49b0\": container with ID starting with 858bee7071c9dcb3a71128974131fbb4b2404cfe479488dc872152ae98ac49b0 not found: ID does not exist" Sep 09 02:57:19.974970 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.974948 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk"] Sep 09 02:57:19.981609 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:19.981589 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f5aebb-849b5bdd64-f27gk"] Sep 09 02:57:20.053852 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:20.053801 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:20.310224 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:20.310146 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="655f79a2-a584-44f2-a615-2cad39a3a608" path="/var/lib/kubelet/pods/655f79a2-a584-44f2-a615-2cad39a3a608/volumes" Sep 09 02:57:21.941114 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:21.941084 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" Sep 09 02:57:25.680378 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:25.680340 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:25.781377 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:25.781319 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:25.972281 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:25.972204 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:35.308950 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:35.308910 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:38.249318 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:38.249286 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:46.468425 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:46.468392 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:57:54.355584 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:57:54.355551 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:58:09.952703 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:09.952668 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:58:10.321041 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:10.320927 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:58:10.826719 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:10.826679 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:58:10.900045 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:10.900015 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:58:12.588314 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:12.588282 2577 scope.go:117] "RemoveContainer" containerID="a2b04f72bc2525cf71a72f90713423a7e5125fbd058434a89ea1f092a8db3f7e" Sep 09 02:58:12.596896 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:12.596873 2577 scope.go:117] "RemoveContainer" containerID="f91939279966ed1b2998eee631e31b984b8f9c0ae0ff60187b20fa2a4895bf40" Sep 09 02:58:12.604763 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:12.604747 2577 scope.go:117] "RemoveContainer" containerID="69a63d9e6da9fbf96e3fa3bbf33ecc2a4b092423ce32ba0140120fcaf8bbfd72" Sep 09 02:58:13.489077 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:13.489023 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:58:13.605323 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:13.605277 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:58:14.616286 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.616254 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd"] Sep 09 02:58:14.616776 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.616753 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="655f79a2-a584-44f2-a615-2cad39a3a608" containerName="maas-api" Sep 09 02:58:14.616967 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.616856 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="655f79a2-a584-44f2-a615-2cad39a3a608" containerName="maas-api" Sep 09 02:58:14.617184 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.617038 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="655f79a2-a584-44f2-a615-2cad39a3a608" containerName="maas-api" Sep 09 02:58:14.620389 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.620371 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" Sep 09 02:58:14.622757 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.622733 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-2b8247-serving-cert\"" Sep 09 02:58:14.632582 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.632558 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd"] Sep 09 02:58:14.650331 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.650306 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:58:14.727967 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.727938 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:58:14.780867 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.780826 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/977c0787-17ba-4277-902c-e0539fd4c719-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd\" (UID: \"977c0787-17ba-4277-902c-e0539fd4c719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" Sep 09 02:58:14.781049 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.780885 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/977c0787-17ba-4277-902c-e0539fd4c719-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd\" (UID: \"977c0787-17ba-4277-902c-e0539fd4c719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" Sep 09 02:58:14.781049 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.780923 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gttm\" (UniqueName: \"kubernetes.io/projected/977c0787-17ba-4277-902c-e0539fd4c719-kube-api-access-6gttm\") pod \"maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd\" (UID: \"977c0787-17ba-4277-902c-e0539fd4c719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" Sep 09 02:58:14.781049 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.780943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/977c0787-17ba-4277-902c-e0539fd4c719-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd\" (UID: \"977c0787-17ba-4277-902c-e0539fd4c719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" Sep 09 02:58:14.823656 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.823629 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:58:14.881955 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.881935 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/977c0787-17ba-4277-902c-e0539fd4c719-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd\" (UID: \"977c0787-17ba-4277-902c-e0539fd4c719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" Sep 09 02:58:14.882089 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.881978 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gttm\" (UniqueName: \"kubernetes.io/projected/977c0787-17ba-4277-902c-e0539fd4c719-kube-api-access-6gttm\") pod \"maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd\" (UID: \"977c0787-17ba-4277-902c-e0539fd4c719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" Sep 09 02:58:14.882089 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.882005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/977c0787-17ba-4277-902c-e0539fd4c719-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd\" (UID: \"977c0787-17ba-4277-902c-e0539fd4c719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" Sep 09 02:58:14.882089 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.882076 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/977c0787-17ba-4277-902c-e0539fd4c719-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd\" (UID: \"977c0787-17ba-4277-902c-e0539fd4c719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" Sep 09 02:58:14.882562 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.882535 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/977c0787-17ba-4277-902c-e0539fd4c719-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd\" (UID: \"977c0787-17ba-4277-902c-e0539fd4c719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" Sep 09 02:58:14.884448 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.884427 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/977c0787-17ba-4277-902c-e0539fd4c719-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd\" (UID: \"977c0787-17ba-4277-902c-e0539fd4c719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" Sep 09 02:58:14.884548 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.884509 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/977c0787-17ba-4277-902c-e0539fd4c719-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd\" (UID: \"977c0787-17ba-4277-902c-e0539fd4c719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" Sep 09 02:58:14.891383 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.891360 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gttm\" (UniqueName: \"kubernetes.io/projected/977c0787-17ba-4277-902c-e0539fd4c719-kube-api-access-6gttm\") pod \"maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd\" (UID: \"977c0787-17ba-4277-902c-e0539fd4c719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" Sep 09 02:58:14.932649 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:14.932618 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" Sep 09 02:58:15.150850 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:15.150729 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:58:15.208519 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:15.208497 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd"] Sep 09 02:58:15.210381 ip-10-0-137-37 kubenswrapper[2577]: W0909 02:58:15.210356 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977c0787_17ba_4277_902c_e0539fd4c719.slice/crio-c8c4003547bc0894e93e834d0fc01339b5bb39a9759f14315dc16474d774bae0 WatchSource:0}: Error finding container c8c4003547bc0894e93e834d0fc01339b5bb39a9759f14315dc16474d774bae0: Status 404 returned error can't find the container with id c8c4003547bc0894e93e834d0fc01339b5bb39a9759f14315dc16474d774bae0 Sep 09 02:58:15.250177 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:15.250143 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" event={"ID":"977c0787-17ba-4277-902c-e0539fd4c719","Type":"ContainerStarted","Data":"c8c4003547bc0894e93e834d0fc01339b5bb39a9759f14315dc16474d774bae0"} Sep 09 02:58:15.252357 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:15.252333 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:58:16.257562 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:16.257522 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" event={"ID":"977c0787-17ba-4277-902c-e0539fd4c719","Type":"ContainerStarted","Data":"4e39e4aef300a29a058be9c8c8f949c32b81f5f966ca3eeaace14eac30322487"} Sep 09 02:58:16.258026 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:16.257589 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" Sep 09 02:58:16.278358 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:16.278309 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" podStartSLOduration=1.8203640490000002 podStartE2EDuration="2.27829308s" totalImagesPullingTime="457.929031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:58:16.275304792 +0000 UTC m=+1684.405546887" watchObservedRunningTime="2026-09-09 02:58:16.27829308 +0000 UTC m=+1684.408535175" Sep 09 02:58:22.267261 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:22.267231 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" Sep 09 02:58:38.582774 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:38.582696 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:58:45.164692 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.164639 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd"] Sep 09 02:58:45.165331 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.164992 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" podUID="977c0787-17ba-4277-902c-e0539fd4c719" containerName="maas-api" containerID="cri-o://4e39e4aef300a29a058be9c8c8f949c32b81f5f966ca3eeaace14eac30322487" gracePeriod=30 Sep 09 02:58:45.199812 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.199788 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:58:45.267643 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.267601 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:58:45.328924 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.328888 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:58:45.385267 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.385237 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:58:45.414348 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.414321 2577 generic.go:356] "Generic (PLEG): container finished" podID="977c0787-17ba-4277-902c-e0539fd4c719" containerID="4e39e4aef300a29a058be9c8c8f949c32b81f5f966ca3eeaace14eac30322487" exitCode=0 Sep 09 02:58:45.414490 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.414367 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" event={"ID":"977c0787-17ba-4277-902c-e0539fd4c719","Type":"ContainerDied","Data":"4e39e4aef300a29a058be9c8c8f949c32b81f5f966ca3eeaace14eac30322487"} Sep 09 02:58:45.414490 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.414403 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" event={"ID":"977c0787-17ba-4277-902c-e0539fd4c719","Type":"ContainerDied","Data":"c8c4003547bc0894e93e834d0fc01339b5bb39a9759f14315dc16474d774bae0"} Sep 09 02:58:45.414490 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.414418 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8c4003547bc0894e93e834d0fc01339b5bb39a9759f14315dc16474d774bae0" Sep 09 02:58:45.422103 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.422050 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" Sep 09 02:58:45.456363 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.456333 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:58:45.575739 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.575701 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/977c0787-17ba-4277-902c-e0539fd4c719-metrics-certs\") pod \"977c0787-17ba-4277-902c-e0539fd4c719\" (UID: \"977c0787-17ba-4277-902c-e0539fd4c719\") " Sep 09 02:58:45.575947 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.575749 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/977c0787-17ba-4277-902c-e0539fd4c719-maas-api-tls\") pod \"977c0787-17ba-4277-902c-e0539fd4c719\" (UID: \"977c0787-17ba-4277-902c-e0539fd4c719\") " Sep 09 02:58:45.575947 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.575866 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gttm\" (UniqueName: \"kubernetes.io/projected/977c0787-17ba-4277-902c-e0539fd4c719-kube-api-access-6gttm\") pod \"977c0787-17ba-4277-902c-e0539fd4c719\" (UID: \"977c0787-17ba-4277-902c-e0539fd4c719\") " Sep 09 02:58:45.575947 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.575894 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/977c0787-17ba-4277-902c-e0539fd4c719-openshift-service-ca\") pod \"977c0787-17ba-4277-902c-e0539fd4c719\" (UID: \"977c0787-17ba-4277-902c-e0539fd4c719\") " Sep 09 02:58:45.576368 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.576336 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/977c0787-17ba-4277-902c-e0539fd4c719-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "977c0787-17ba-4277-902c-e0539fd4c719" (UID: "977c0787-17ba-4277-902c-e0539fd4c719"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:58:45.578025 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.577996 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977c0787-17ba-4277-902c-e0539fd4c719-kube-api-access-6gttm" (OuterVolumeSpecName: "kube-api-access-6gttm") pod "977c0787-17ba-4277-902c-e0539fd4c719" (UID: "977c0787-17ba-4277-902c-e0539fd4c719"). InnerVolumeSpecName "kube-api-access-6gttm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:58:45.578112 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.578062 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/977c0787-17ba-4277-902c-e0539fd4c719-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "977c0787-17ba-4277-902c-e0539fd4c719" (UID: "977c0787-17ba-4277-902c-e0539fd4c719"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:58:45.578154 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.578136 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/977c0787-17ba-4277-902c-e0539fd4c719-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "977c0787-17ba-4277-902c-e0539fd4c719" (UID: "977c0787-17ba-4277-902c-e0539fd4c719"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:58:45.677544 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.677461 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/977c0787-17ba-4277-902c-e0539fd4c719-metrics-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:58:45.677544 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.677489 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/977c0787-17ba-4277-902c-e0539fd4c719-maas-api-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:58:45.677544 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.677504 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gttm\" (UniqueName: \"kubernetes.io/projected/977c0787-17ba-4277-902c-e0539fd4c719-kube-api-access-6gttm\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:58:45.677544 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.677517 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/977c0787-17ba-4277-902c-e0539fd4c719-openshift-service-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:58:45.783367 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:45.783331 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:58:46.419356 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:46.419309 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd" Sep 09 02:58:46.438662 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:46.438630 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd"] Sep 09 02:58:46.442225 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:46.442200 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2b8247-846c748b8b-td7wd"] Sep 09 02:58:48.310405 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:48.310370 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="977c0787-17ba-4277-902c-e0539fd4c719" path="/var/lib/kubelet/pods/977c0787-17ba-4277-902c-e0539fd4c719/volumes" Sep 09 02:58:51.684864 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:51.684758 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:58:51.745423 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:51.745391 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:58:52.052336 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:52.052244 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:58:56.244437 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:58:56.244402 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:59:11.597443 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:11.597410 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh"] Sep 09 02:59:11.597866 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:11.597616 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" podUID="38c3ad74-d112-4a2f-a7e1-8fc14b02853b" containerName="maas-api" containerID="cri-o://af97c7fcd2104d4d05a5b6c75633fc0716e144a4cba006c631deacb579873faf" gracePeriod=30 Sep 09 02:59:11.627389 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:11.627357 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:59:11.687766 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:11.687735 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:59:11.823609 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:11.823577 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:59:11.855579 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:11.855528 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" Sep 09 02:59:11.876567 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:11.876525 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:59:11.933035 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:11.933003 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-metrics-certs\") pod \"38c3ad74-d112-4a2f-a7e1-8fc14b02853b\" (UID: \"38c3ad74-d112-4a2f-a7e1-8fc14b02853b\") " Sep 09 02:59:11.933235 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:11.933047 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-openshift-service-ca\") pod \"38c3ad74-d112-4a2f-a7e1-8fc14b02853b\" (UID: \"38c3ad74-d112-4a2f-a7e1-8fc14b02853b\") " Sep 09 02:59:11.933235 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:11.933101 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xl5ts\" (UniqueName: \"kubernetes.io/projected/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-kube-api-access-xl5ts\") pod \"38c3ad74-d112-4a2f-a7e1-8fc14b02853b\" (UID: \"38c3ad74-d112-4a2f-a7e1-8fc14b02853b\") " Sep 09 02:59:11.933235 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:11.933200 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-maas-api-tls\") pod \"38c3ad74-d112-4a2f-a7e1-8fc14b02853b\" (UID: \"38c3ad74-d112-4a2f-a7e1-8fc14b02853b\") " Sep 09 02:59:11.933455 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:11.933431 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "38c3ad74-d112-4a2f-a7e1-8fc14b02853b" (UID: "38c3ad74-d112-4a2f-a7e1-8fc14b02853b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:59:11.935261 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:11.935232 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-kube-api-access-xl5ts" (OuterVolumeSpecName: "kube-api-access-xl5ts") pod "38c3ad74-d112-4a2f-a7e1-8fc14b02853b" (UID: "38c3ad74-d112-4a2f-a7e1-8fc14b02853b"). InnerVolumeSpecName "kube-api-access-xl5ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:59:11.935261 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:11.935240 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "38c3ad74-d112-4a2f-a7e1-8fc14b02853b" (UID: "38c3ad74-d112-4a2f-a7e1-8fc14b02853b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:59:11.935433 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:11.935365 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "38c3ad74-d112-4a2f-a7e1-8fc14b02853b" (UID: "38c3ad74-d112-4a2f-a7e1-8fc14b02853b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:59:12.034237 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:12.034207 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-maas-api-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:59:12.034237 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:12.034232 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-metrics-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:59:12.034237 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:12.034240 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-openshift-service-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:59:12.034458 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:12.034250 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl5ts\" (UniqueName: \"kubernetes.io/projected/38c3ad74-d112-4a2f-a7e1-8fc14b02853b-kube-api-access-xl5ts\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 02:59:12.262263 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:12.262228 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:59:12.558179 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:12.558084 2577 generic.go:356] "Generic (PLEG): container finished" podID="38c3ad74-d112-4a2f-a7e1-8fc14b02853b" containerID="af97c7fcd2104d4d05a5b6c75633fc0716e144a4cba006c631deacb579873faf" exitCode=0 Sep 09 02:59:12.558179 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:12.558134 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" Sep 09 02:59:12.558378 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:12.558169 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" event={"ID":"38c3ad74-d112-4a2f-a7e1-8fc14b02853b","Type":"ContainerDied","Data":"af97c7fcd2104d4d05a5b6c75633fc0716e144a4cba006c631deacb579873faf"} Sep 09 02:59:12.558378 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:12.558250 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh" event={"ID":"38c3ad74-d112-4a2f-a7e1-8fc14b02853b","Type":"ContainerDied","Data":"de5a218ddcece4d1acb4f625218dc659bd736ce9f3331dc88e5e45cdab6e7ee1"} Sep 09 02:59:12.558378 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:12.558277 2577 scope.go:117] "RemoveContainer" containerID="af97c7fcd2104d4d05a5b6c75633fc0716e144a4cba006c631deacb579873faf" Sep 09 02:59:12.567418 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:12.567398 2577 scope.go:117] "RemoveContainer" containerID="af97c7fcd2104d4d05a5b6c75633fc0716e144a4cba006c631deacb579873faf" Sep 09 02:59:12.567690 ip-10-0-137-37 kubenswrapper[2577]: E0909 02:59:12.567670 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af97c7fcd2104d4d05a5b6c75633fc0716e144a4cba006c631deacb579873faf\": container with ID starting with af97c7fcd2104d4d05a5b6c75633fc0716e144a4cba006c631deacb579873faf not found: ID does not exist" containerID="af97c7fcd2104d4d05a5b6c75633fc0716e144a4cba006c631deacb579873faf" Sep 09 02:59:12.567743 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:12.567697 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af97c7fcd2104d4d05a5b6c75633fc0716e144a4cba006c631deacb579873faf"} err="failed to get container status \"af97c7fcd2104d4d05a5b6c75633fc0716e144a4cba006c631deacb579873faf\": rpc error: code = NotFound desc = could not find container \"af97c7fcd2104d4d05a5b6c75633fc0716e144a4cba006c631deacb579873faf\": container with ID starting with af97c7fcd2104d4d05a5b6c75633fc0716e144a4cba006c631deacb579873faf not found: ID does not exist" Sep 09 02:59:12.574256 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:12.574232 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh"] Sep 09 02:59:12.577781 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:12.577756 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2085ee-5484646457-brxgh"] Sep 09 02:59:14.310108 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:14.310076 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38c3ad74-d112-4a2f-a7e1-8fc14b02853b" path="/var/lib/kubelet/pods/38c3ad74-d112-4a2f-a7e1-8fc14b02853b/volumes" Sep 09 02:59:17.293381 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:17.293348 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:59:23.728941 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:23.728909 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:59:54.039535 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:54.039498 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:59:54.318038 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:54.317958 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:59:59.884473 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:59.884440 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 02:59:59.951806 ip-10-0-137-37 kubenswrapper[2577]: I0909 02:59:59.951772 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:00.203938 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:00.203860 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:00.742754 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:00.742723 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:11.066812 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.066735 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc"] Sep 09 03:00:11.067269 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.066967 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" podUID="18778870-20d2-4e85-b4c8-52dbd1933b24" containerName="maas-api" containerID="cri-o://f4601e07b45a14fd2430fe0f1621321a6530dc0b8b22a28e0ddabbd1d1ae2f61" gracePeriod=30 Sep 09 03:00:11.101468 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.101437 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:11.182660 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.182621 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:11.300092 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.300059 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:11.315889 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.315864 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" Sep 09 03:00:11.411242 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.411207 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18778870-20d2-4e85-b4c8-52dbd1933b24-metrics-certs\") pod \"18778870-20d2-4e85-b4c8-52dbd1933b24\" (UID: \"18778870-20d2-4e85-b4c8-52dbd1933b24\") " Sep 09 03:00:11.411242 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.411246 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-855vf\" (UniqueName: \"kubernetes.io/projected/18778870-20d2-4e85-b4c8-52dbd1933b24-kube-api-access-855vf\") pod \"18778870-20d2-4e85-b4c8-52dbd1933b24\" (UID: \"18778870-20d2-4e85-b4c8-52dbd1933b24\") " Sep 09 03:00:11.411505 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.411327 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18778870-20d2-4e85-b4c8-52dbd1933b24-openshift-service-ca\") pod \"18778870-20d2-4e85-b4c8-52dbd1933b24\" (UID: \"18778870-20d2-4e85-b4c8-52dbd1933b24\") " Sep 09 03:00:11.411505 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.411360 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18778870-20d2-4e85-b4c8-52dbd1933b24-maas-api-tls\") pod \"18778870-20d2-4e85-b4c8-52dbd1933b24\" (UID: \"18778870-20d2-4e85-b4c8-52dbd1933b24\") " Sep 09 03:00:11.411717 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.411689 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18778870-20d2-4e85-b4c8-52dbd1933b24-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "18778870-20d2-4e85-b4c8-52dbd1933b24" (UID: "18778870-20d2-4e85-b4c8-52dbd1933b24"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 03:00:11.413355 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.413326 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18778870-20d2-4e85-b4c8-52dbd1933b24-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "18778870-20d2-4e85-b4c8-52dbd1933b24" (UID: "18778870-20d2-4e85-b4c8-52dbd1933b24"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 03:00:11.413479 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.413395 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18778870-20d2-4e85-b4c8-52dbd1933b24-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "18778870-20d2-4e85-b4c8-52dbd1933b24" (UID: "18778870-20d2-4e85-b4c8-52dbd1933b24"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 03:00:11.413526 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.413479 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18778870-20d2-4e85-b4c8-52dbd1933b24-kube-api-access-855vf" (OuterVolumeSpecName: "kube-api-access-855vf") pod "18778870-20d2-4e85-b4c8-52dbd1933b24" (UID: "18778870-20d2-4e85-b4c8-52dbd1933b24"). InnerVolumeSpecName "kube-api-access-855vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 03:00:11.425408 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.425386 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:11.512338 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.512310 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18778870-20d2-4e85-b4c8-52dbd1933b24-metrics-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:11.512338 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.512334 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-855vf\" (UniqueName: \"kubernetes.io/projected/18778870-20d2-4e85-b4c8-52dbd1933b24-kube-api-access-855vf\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:11.512338 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.512344 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18778870-20d2-4e85-b4c8-52dbd1933b24-openshift-service-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:11.512583 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.512352 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18778870-20d2-4e85-b4c8-52dbd1933b24-maas-api-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:11.717440 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.717357 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:11.875407 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.875374 2577 generic.go:356] "Generic (PLEG): container finished" podID="18778870-20d2-4e85-b4c8-52dbd1933b24" containerID="f4601e07b45a14fd2430fe0f1621321a6530dc0b8b22a28e0ddabbd1d1ae2f61" exitCode=0 Sep 09 03:00:11.875598 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.875426 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" Sep 09 03:00:11.875598 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.875462 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" event={"ID":"18778870-20d2-4e85-b4c8-52dbd1933b24","Type":"ContainerDied","Data":"f4601e07b45a14fd2430fe0f1621321a6530dc0b8b22a28e0ddabbd1d1ae2f61"} Sep 09 03:00:11.875598 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.875514 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc" event={"ID":"18778870-20d2-4e85-b4c8-52dbd1933b24","Type":"ContainerDied","Data":"46a73e5ef4db9985a5613d2094083585e4f24b9e47788852c85fcfaf0dee1ac7"} Sep 09 03:00:11.875598 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.875537 2577 scope.go:117] "RemoveContainer" containerID="f4601e07b45a14fd2430fe0f1621321a6530dc0b8b22a28e0ddabbd1d1ae2f61" Sep 09 03:00:11.885783 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.885766 2577 scope.go:117] "RemoveContainer" containerID="f4601e07b45a14fd2430fe0f1621321a6530dc0b8b22a28e0ddabbd1d1ae2f61" Sep 09 03:00:11.886117 ip-10-0-137-37 kubenswrapper[2577]: E0909 03:00:11.886097 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4601e07b45a14fd2430fe0f1621321a6530dc0b8b22a28e0ddabbd1d1ae2f61\": container with ID starting with f4601e07b45a14fd2430fe0f1621321a6530dc0b8b22a28e0ddabbd1d1ae2f61 not found: ID does not exist" containerID="f4601e07b45a14fd2430fe0f1621321a6530dc0b8b22a28e0ddabbd1d1ae2f61" Sep 09 03:00:11.886181 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.886126 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4601e07b45a14fd2430fe0f1621321a6530dc0b8b22a28e0ddabbd1d1ae2f61"} err="failed to get container status \"f4601e07b45a14fd2430fe0f1621321a6530dc0b8b22a28e0ddabbd1d1ae2f61\": rpc error: code = NotFound desc = could not find container \"f4601e07b45a14fd2430fe0f1621321a6530dc0b8b22a28e0ddabbd1d1ae2f61\": container with ID starting with f4601e07b45a14fd2430fe0f1621321a6530dc0b8b22a28e0ddabbd1d1ae2f61 not found: ID does not exist" Sep 09 03:00:11.897071 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.897040 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc"] Sep 09 03:00:11.901961 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:11.901934 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-621ac9-86778dd6c9-7l9cc"] Sep 09 03:00:12.311604 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:12.311567 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18778870-20d2-4e85-b4c8-52dbd1933b24" path="/var/lib/kubelet/pods/18778870-20d2-4e85-b4c8-52dbd1933b24/volumes" Sep 09 03:00:12.452776 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:12.452746 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/2.log" Sep 09 03:00:12.453577 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:12.453554 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/2.log" Sep 09 03:00:17.690420 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:17.690384 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:17.752836 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:17.752787 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:17.989477 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:17.989386 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:18.559318 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:18.559284 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:29.022090 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.022047 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx"] Sep 09 03:00:29.022665 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.022352 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" podUID="b0bf842c-206e-40ca-9d32-602d6d3fcd9b" containerName="maas-api" containerID="cri-o://72b15296738fb251ea9e5b65c675f9b1416663b4d4236252d5bedd4db4c08702" gracePeriod=30 Sep 09 03:00:29.044189 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.044164 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:29.113596 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.113566 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:29.176548 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.176511 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:29.236989 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.236961 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:29.270999 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.270977 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" Sep 09 03:00:29.292669 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.292577 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:29.382037 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.382003 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjgk5\" (UniqueName: \"kubernetes.io/projected/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-kube-api-access-qjgk5\") pod \"b0bf842c-206e-40ca-9d32-602d6d3fcd9b\" (UID: \"b0bf842c-206e-40ca-9d32-602d6d3fcd9b\") " Sep 09 03:00:29.382202 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.382078 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-metrics-certs\") pod \"b0bf842c-206e-40ca-9d32-602d6d3fcd9b\" (UID: \"b0bf842c-206e-40ca-9d32-602d6d3fcd9b\") " Sep 09 03:00:29.382202 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.382148 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-maas-api-tls\") pod \"b0bf842c-206e-40ca-9d32-602d6d3fcd9b\" (UID: \"b0bf842c-206e-40ca-9d32-602d6d3fcd9b\") " Sep 09 03:00:29.382202 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.382189 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-openshift-service-ca\") pod \"b0bf842c-206e-40ca-9d32-602d6d3fcd9b\" (UID: \"b0bf842c-206e-40ca-9d32-602d6d3fcd9b\") " Sep 09 03:00:29.382652 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.382620 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b0bf842c-206e-40ca-9d32-602d6d3fcd9b" (UID: "b0bf842c-206e-40ca-9d32-602d6d3fcd9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 03:00:29.384406 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.384372 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b0bf842c-206e-40ca-9d32-602d6d3fcd9b" (UID: "b0bf842c-206e-40ca-9d32-602d6d3fcd9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 03:00:29.384666 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.384639 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b0bf842c-206e-40ca-9d32-602d6d3fcd9b" (UID: "b0bf842c-206e-40ca-9d32-602d6d3fcd9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 03:00:29.384766 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.384673 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-kube-api-access-qjgk5" (OuterVolumeSpecName: "kube-api-access-qjgk5") pod "b0bf842c-206e-40ca-9d32-602d6d3fcd9b" (UID: "b0bf842c-206e-40ca-9d32-602d6d3fcd9b"). InnerVolumeSpecName "kube-api-access-qjgk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 03:00:29.483520 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.483467 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjgk5\" (UniqueName: \"kubernetes.io/projected/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-kube-api-access-qjgk5\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:29.483520 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.483514 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-metrics-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:29.483520 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.483524 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-maas-api-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:29.483520 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.483533 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bf842c-206e-40ca-9d32-602d6d3fcd9b-openshift-service-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:29.644413 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.644379 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:29.973875 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.973764 2577 generic.go:356] "Generic (PLEG): container finished" podID="b0bf842c-206e-40ca-9d32-602d6d3fcd9b" containerID="72b15296738fb251ea9e5b65c675f9b1416663b4d4236252d5bedd4db4c08702" exitCode=0 Sep 09 03:00:29.973875 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.973811 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" Sep 09 03:00:29.973875 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.973857 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" event={"ID":"b0bf842c-206e-40ca-9d32-602d6d3fcd9b","Type":"ContainerDied","Data":"72b15296738fb251ea9e5b65c675f9b1416663b4d4236252d5bedd4db4c08702"} Sep 09 03:00:29.974137 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.973916 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx" event={"ID":"b0bf842c-206e-40ca-9d32-602d6d3fcd9b","Type":"ContainerDied","Data":"d85accf11f88edb7f56cddc94a1c85ae16fa1e9726af025b4d4779b04ed69b3c"} Sep 09 03:00:29.974137 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.973947 2577 scope.go:117] "RemoveContainer" containerID="72b15296738fb251ea9e5b65c675f9b1416663b4d4236252d5bedd4db4c08702" Sep 09 03:00:29.984112 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.984093 2577 scope.go:117] "RemoveContainer" containerID="72b15296738fb251ea9e5b65c675f9b1416663b4d4236252d5bedd4db4c08702" Sep 09 03:00:29.984373 ip-10-0-137-37 kubenswrapper[2577]: E0909 03:00:29.984352 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b15296738fb251ea9e5b65c675f9b1416663b4d4236252d5bedd4db4c08702\": container with ID starting with 72b15296738fb251ea9e5b65c675f9b1416663b4d4236252d5bedd4db4c08702 not found: ID does not exist" containerID="72b15296738fb251ea9e5b65c675f9b1416663b4d4236252d5bedd4db4c08702" Sep 09 03:00:29.984421 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.984382 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b15296738fb251ea9e5b65c675f9b1416663b4d4236252d5bedd4db4c08702"} err="failed to get container status \"72b15296738fb251ea9e5b65c675f9b1416663b4d4236252d5bedd4db4c08702\": rpc error: code = NotFound desc = could not find container \"72b15296738fb251ea9e5b65c675f9b1416663b4d4236252d5bedd4db4c08702\": container with ID starting with 72b15296738fb251ea9e5b65c675f9b1416663b4d4236252d5bedd4db4c08702 not found: ID does not exist" Sep 09 03:00:29.996181 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:29.996156 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx"] Sep 09 03:00:30.002460 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:30.002427 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-56dbc0-6b89f69879-dhmpx"] Sep 09 03:00:30.310245 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:30.310168 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0bf842c-206e-40ca-9d32-602d6d3fcd9b" path="/var/lib/kubelet/pods/b0bf842c-206e-40ca-9d32-602d6d3fcd9b/volumes" Sep 09 03:00:35.569488 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:35.569455 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:35.633372 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:35.633310 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:35.821530 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:35.821439 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f"] Sep 09 03:00:35.821769 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:35.821734 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" podUID="e4aeb107-020b-4588-943d-ce3f93f8d571" containerName="istio-proxy" containerID="cri-o://cddd455f2ee888d46a68b4ec7fbacf74246f4f9871a81ab67799ca79ad2d7610" gracePeriod=30 Sep 09 03:00:35.848064 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:35.848033 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:36.318781 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:36.318742 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:38.548055 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:38.548014 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 09 03:00:38.548441 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:38.548079 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" podUID="e4aeb107-020b-4588-943d-ce3f93f8d571" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 03:00:41.038449 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.038419 2577 generic.go:356] "Generic (PLEG): container finished" podID="e4aeb107-020b-4588-943d-ce3f93f8d571" containerID="cddd455f2ee888d46a68b4ec7fbacf74246f4f9871a81ab67799ca79ad2d7610" exitCode=0 Sep 09 03:00:41.038934 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.038495 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" event={"ID":"e4aeb107-020b-4588-943d-ce3f93f8d571","Type":"ContainerDied","Data":"cddd455f2ee888d46a68b4ec7fbacf74246f4f9871a81ab67799ca79ad2d7610"} Sep 09 03:00:41.038934 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.038569 2577 scope.go:117] "RemoveContainer" containerID="86e89e73bf72f9a49bf615436dc7fe1679054c2b185bdce3615768f867d185b4" Sep 09 03:00:41.073598 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.073577 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 03:00:41.197896 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.197865 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4aeb107-020b-4588-943d-ce3f93f8d571-istiod-ca-cert\") pod \"e4aeb107-020b-4588-943d-ce3f93f8d571\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " Sep 09 03:00:41.198109 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.197905 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-workload-socket\") pod \"e4aeb107-020b-4588-943d-ce3f93f8d571\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " Sep 09 03:00:41.198109 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.197925 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-credential-socket\") pod \"e4aeb107-020b-4588-943d-ce3f93f8d571\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " Sep 09 03:00:41.198109 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.198001 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-workload-certs\") pod \"e4aeb107-020b-4588-943d-ce3f93f8d571\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " Sep 09 03:00:41.198109 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.198016 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-data\") pod \"e4aeb107-020b-4588-943d-ce3f93f8d571\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " Sep 09 03:00:41.198109 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.198048 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-envoy\") pod \"e4aeb107-020b-4588-943d-ce3f93f8d571\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " Sep 09 03:00:41.198109 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.198077 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddntv\" (UniqueName: \"kubernetes.io/projected/e4aeb107-020b-4588-943d-ce3f93f8d571-kube-api-access-ddntv\") pod \"e4aeb107-020b-4588-943d-ce3f93f8d571\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " Sep 09 03:00:41.198109 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.198101 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-podinfo\") pod \"e4aeb107-020b-4588-943d-ce3f93f8d571\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " Sep 09 03:00:41.198427 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.198117 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-token\") pod \"e4aeb107-020b-4588-943d-ce3f93f8d571\" (UID: \"e4aeb107-020b-4588-943d-ce3f93f8d571\") " Sep 09 03:00:41.198637 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.198609 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4aeb107-020b-4588-943d-ce3f93f8d571-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e4aeb107-020b-4588-943d-ce3f93f8d571" (UID: "e4aeb107-020b-4588-943d-ce3f93f8d571"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 03:00:41.198725 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.198685 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e4aeb107-020b-4588-943d-ce3f93f8d571" (UID: "e4aeb107-020b-4588-943d-ce3f93f8d571"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 03:00:41.198908 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.198864 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e4aeb107-020b-4588-943d-ce3f93f8d571" (UID: "e4aeb107-020b-4588-943d-ce3f93f8d571"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 03:00:41.199200 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.199007 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e4aeb107-020b-4588-943d-ce3f93f8d571" (UID: "e4aeb107-020b-4588-943d-ce3f93f8d571"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 03:00:41.199377 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.199357 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-data" (OuterVolumeSpecName: "istio-data") pod "e4aeb107-020b-4588-943d-ce3f93f8d571" (UID: "e4aeb107-020b-4588-943d-ce3f93f8d571"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 03:00:41.202763 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.201600 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e4aeb107-020b-4588-943d-ce3f93f8d571" (UID: "e4aeb107-020b-4588-943d-ce3f93f8d571"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 03:00:41.208325 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.208295 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e4aeb107-020b-4588-943d-ce3f93f8d571" (UID: "e4aeb107-020b-4588-943d-ce3f93f8d571"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 03:00:41.209756 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.209724 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4aeb107-020b-4588-943d-ce3f93f8d571-kube-api-access-ddntv" (OuterVolumeSpecName: "kube-api-access-ddntv") pod "e4aeb107-020b-4588-943d-ce3f93f8d571" (UID: "e4aeb107-020b-4588-943d-ce3f93f8d571"). InnerVolumeSpecName "kube-api-access-ddntv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 03:00:41.210210 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.210069 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-token" (OuterVolumeSpecName: "istio-token") pod "e4aeb107-020b-4588-943d-ce3f93f8d571" (UID: "e4aeb107-020b-4588-943d-ce3f93f8d571"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 03:00:41.299665 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.299629 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-workload-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:41.299665 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.299659 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-data\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:41.299665 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.299667 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-envoy\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:41.299940 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.299697 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddntv\" (UniqueName: \"kubernetes.io/projected/e4aeb107-020b-4588-943d-ce3f93f8d571-kube-api-access-ddntv\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:41.299940 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.299708 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-podinfo\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:41.299940 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.299716 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4aeb107-020b-4588-943d-ce3f93f8d571-istio-token\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:41.299940 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.299725 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4aeb107-020b-4588-943d-ce3f93f8d571-istiod-ca-cert\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:41.299940 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.299732 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-workload-socket\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:41.299940 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:41.299740 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4aeb107-020b-4588-943d-ce3f93f8d571-credential-socket\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:42.045115 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:42.045089 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" Sep 09 03:00:42.045115 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:42.045103 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f" event={"ID":"e4aeb107-020b-4588-943d-ce3f93f8d571","Type":"ContainerDied","Data":"9de84e4e63930cf2bdee180ad4433440e5120bc8ac27c340ad8cb4882e51061b"} Sep 09 03:00:42.045683 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:42.045170 2577 scope.go:117] "RemoveContainer" containerID="cddd455f2ee888d46a68b4ec7fbacf74246f4f9871a81ab67799ca79ad2d7610" Sep 09 03:00:42.067081 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:42.067043 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f"] Sep 09 03:00:42.072751 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:42.072722 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-56dbc0-openshift-default-5fdf4dd656-gkd5f"] Sep 09 03:00:42.313625 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:42.313545 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4aeb107-020b-4588-943d-ce3f93f8d571" path="/var/lib/kubelet/pods/e4aeb107-020b-4588-943d-ce3f93f8d571/volumes" Sep 09 03:00:46.633180 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:46.633143 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb"] Sep 09 03:00:46.633668 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:46.633382 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" podUID="2f4d5ed5-0912-4bba-b926-4614ea6a0776" containerName="maas-api" containerID="cri-o://ac5b9b6d7b4cda603fc6b554a8158412f883c69fb4cc407d1084fbd87ab81fb7" gracePeriod=30 Sep 09 03:00:46.663151 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:46.663099 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:46.689197 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:46.689161 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8"] Sep 09 03:00:46.689458 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:46.689430 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8" podUID="d5be666a-6315-45f7-8593-f6474ac6435b" containerName="payload-processing" containerID="cri-o://df472d3d01f11a50ccf95ddfcfe74c1a2217b894454c4317b7d4f0b782b1a6ee" gracePeriod=30 Sep 09 03:00:46.935007 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:46.934982 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" Sep 09 03:00:46.962751 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:46.962724 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8" Sep 09 03:00:47.057604 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.057572 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxf5b\" (UniqueName: \"kubernetes.io/projected/2f4d5ed5-0912-4bba-b926-4614ea6a0776-kube-api-access-lxf5b\") pod \"2f4d5ed5-0912-4bba-b926-4614ea6a0776\" (UID: \"2f4d5ed5-0912-4bba-b926-4614ea6a0776\") " Sep 09 03:00:47.057800 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.057619 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4d5ed5-0912-4bba-b926-4614ea6a0776-openshift-service-ca\") pod \"2f4d5ed5-0912-4bba-b926-4614ea6a0776\" (UID: \"2f4d5ed5-0912-4bba-b926-4614ea6a0776\") " Sep 09 03:00:47.057800 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.057700 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f4d5ed5-0912-4bba-b926-4614ea6a0776-metrics-certs\") pod \"2f4d5ed5-0912-4bba-b926-4614ea6a0776\" (UID: \"2f4d5ed5-0912-4bba-b926-4614ea6a0776\") " Sep 09 03:00:47.057800 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.057752 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4d5ed5-0912-4bba-b926-4614ea6a0776-maas-api-tls\") pod \"2f4d5ed5-0912-4bba-b926-4614ea6a0776\" (UID: \"2f4d5ed5-0912-4bba-b926-4614ea6a0776\") " Sep 09 03:00:47.058062 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.058040 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4d5ed5-0912-4bba-b926-4614ea6a0776-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2f4d5ed5-0912-4bba-b926-4614ea6a0776" (UID: "2f4d5ed5-0912-4bba-b926-4614ea6a0776"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 03:00:47.058131 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.058110 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4d5ed5-0912-4bba-b926-4614ea6a0776-openshift-service-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:47.059843 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.059791 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f4d5ed5-0912-4bba-b926-4614ea6a0776-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2f4d5ed5-0912-4bba-b926-4614ea6a0776" (UID: "2f4d5ed5-0912-4bba-b926-4614ea6a0776"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 03:00:47.059843 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.059831 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f4d5ed5-0912-4bba-b926-4614ea6a0776-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2f4d5ed5-0912-4bba-b926-4614ea6a0776" (UID: "2f4d5ed5-0912-4bba-b926-4614ea6a0776"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 03:00:47.059990 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.059873 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4d5ed5-0912-4bba-b926-4614ea6a0776-kube-api-access-lxf5b" (OuterVolumeSpecName: "kube-api-access-lxf5b") pod "2f4d5ed5-0912-4bba-b926-4614ea6a0776" (UID: "2f4d5ed5-0912-4bba-b926-4614ea6a0776"). InnerVolumeSpecName "kube-api-access-lxf5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 03:00:47.079339 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.079295 2577 generic.go:356] "Generic (PLEG): container finished" podID="2f4d5ed5-0912-4bba-b926-4614ea6a0776" containerID="ac5b9b6d7b4cda603fc6b554a8158412f883c69fb4cc407d1084fbd87ab81fb7" exitCode=0 Sep 09 03:00:47.079540 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.079482 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" event={"ID":"2f4d5ed5-0912-4bba-b926-4614ea6a0776","Type":"ContainerDied","Data":"ac5b9b6d7b4cda603fc6b554a8158412f883c69fb4cc407d1084fbd87ab81fb7"} Sep 09 03:00:47.079613 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.079575 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" event={"ID":"2f4d5ed5-0912-4bba-b926-4614ea6a0776","Type":"ContainerDied","Data":"77a8450c83a9b75915e5f80f7ed91af6bb05e517844d045f6dc2b07e64bc49db"} Sep 09 03:00:47.079676 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.079621 2577 scope.go:117] "RemoveContainer" containerID="ac5b9b6d7b4cda603fc6b554a8158412f883c69fb4cc407d1084fbd87ab81fb7" Sep 09 03:00:47.079862 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.079844 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb" Sep 09 03:00:47.084113 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.084087 2577 generic.go:356] "Generic (PLEG): container finished" podID="d5be666a-6315-45f7-8593-f6474ac6435b" containerID="df472d3d01f11a50ccf95ddfcfe74c1a2217b894454c4317b7d4f0b782b1a6ee" exitCode=0 Sep 09 03:00:47.084233 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.084144 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8" Sep 09 03:00:47.084233 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.084178 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8" event={"ID":"d5be666a-6315-45f7-8593-f6474ac6435b","Type":"ContainerDied","Data":"df472d3d01f11a50ccf95ddfcfe74c1a2217b894454c4317b7d4f0b782b1a6ee"} Sep 09 03:00:47.084233 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.084231 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8" event={"ID":"d5be666a-6315-45f7-8593-f6474ac6435b","Type":"ContainerDied","Data":"3ba6be3b0ab5337181657974c2f1e5335893b57f3cd3a4fc702063c33df008fa"} Sep 09 03:00:47.093375 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.093359 2577 scope.go:117] "RemoveContainer" containerID="ac5b9b6d7b4cda603fc6b554a8158412f883c69fb4cc407d1084fbd87ab81fb7" Sep 09 03:00:47.093628 ip-10-0-137-37 kubenswrapper[2577]: E0909 03:00:47.093610 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac5b9b6d7b4cda603fc6b554a8158412f883c69fb4cc407d1084fbd87ab81fb7\": container with ID starting with ac5b9b6d7b4cda603fc6b554a8158412f883c69fb4cc407d1084fbd87ab81fb7 not found: ID does not exist" containerID="ac5b9b6d7b4cda603fc6b554a8158412f883c69fb4cc407d1084fbd87ab81fb7" Sep 09 03:00:47.093683 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.093635 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac5b9b6d7b4cda603fc6b554a8158412f883c69fb4cc407d1084fbd87ab81fb7"} err="failed to get container status \"ac5b9b6d7b4cda603fc6b554a8158412f883c69fb4cc407d1084fbd87ab81fb7\": rpc error: code = NotFound desc = could not find container \"ac5b9b6d7b4cda603fc6b554a8158412f883c69fb4cc407d1084fbd87ab81fb7\": container with ID starting with ac5b9b6d7b4cda603fc6b554a8158412f883c69fb4cc407d1084fbd87ab81fb7 not found: ID does not exist" Sep 09 03:00:47.093683 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.093651 2577 scope.go:117] "RemoveContainer" containerID="df472d3d01f11a50ccf95ddfcfe74c1a2217b894454c4317b7d4f0b782b1a6ee" Sep 09 03:00:47.101812 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.101787 2577 scope.go:117] "RemoveContainer" containerID="df472d3d01f11a50ccf95ddfcfe74c1a2217b894454c4317b7d4f0b782b1a6ee" Sep 09 03:00:47.102139 ip-10-0-137-37 kubenswrapper[2577]: E0909 03:00:47.102111 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df472d3d01f11a50ccf95ddfcfe74c1a2217b894454c4317b7d4f0b782b1a6ee\": container with ID starting with df472d3d01f11a50ccf95ddfcfe74c1a2217b894454c4317b7d4f0b782b1a6ee not found: ID does not exist" containerID="df472d3d01f11a50ccf95ddfcfe74c1a2217b894454c4317b7d4f0b782b1a6ee" Sep 09 03:00:47.102221 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.102146 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df472d3d01f11a50ccf95ddfcfe74c1a2217b894454c4317b7d4f0b782b1a6ee"} err="failed to get container status \"df472d3d01f11a50ccf95ddfcfe74c1a2217b894454c4317b7d4f0b782b1a6ee\": rpc error: code = NotFound desc = could not find container \"df472d3d01f11a50ccf95ddfcfe74c1a2217b894454c4317b7d4f0b782b1a6ee\": container with ID starting with df472d3d01f11a50ccf95ddfcfe74c1a2217b894454c4317b7d4f0b782b1a6ee not found: ID does not exist" Sep 09 03:00:47.104548 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.104525 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb"] Sep 09 03:00:47.109568 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.109545 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c173ad-6777cbf795-kptpb"] Sep 09 03:00:47.158454 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.158426 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwklg\" (UniqueName: \"kubernetes.io/projected/d5be666a-6315-45f7-8593-f6474ac6435b-kube-api-access-dwklg\") pod \"d5be666a-6315-45f7-8593-f6474ac6435b\" (UID: \"d5be666a-6315-45f7-8593-f6474ac6435b\") " Sep 09 03:00:47.158643 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.158628 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5be666a-6315-45f7-8593-f6474ac6435b-plugins-config-volume\") pod \"d5be666a-6315-45f7-8593-f6474ac6435b\" (UID: \"d5be666a-6315-45f7-8593-f6474ac6435b\") " Sep 09 03:00:47.158878 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.158864 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f4d5ed5-0912-4bba-b926-4614ea6a0776-metrics-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:47.158977 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.158878 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4d5ed5-0912-4bba-b926-4614ea6a0776-maas-api-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:47.158977 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.158892 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxf5b\" (UniqueName: \"kubernetes.io/projected/2f4d5ed5-0912-4bba-b926-4614ea6a0776-kube-api-access-lxf5b\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:47.158977 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.158933 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5be666a-6315-45f7-8593-f6474ac6435b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d5be666a-6315-45f7-8593-f6474ac6435b" (UID: "d5be666a-6315-45f7-8593-f6474ac6435b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 03:00:47.160653 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.160626 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5be666a-6315-45f7-8593-f6474ac6435b-kube-api-access-dwklg" (OuterVolumeSpecName: "kube-api-access-dwklg") pod "d5be666a-6315-45f7-8593-f6474ac6435b" (UID: "d5be666a-6315-45f7-8593-f6474ac6435b"). InnerVolumeSpecName "kube-api-access-dwklg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 03:00:47.259936 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.259891 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5be666a-6315-45f7-8593-f6474ac6435b-plugins-config-volume\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:47.259936 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.259928 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwklg\" (UniqueName: \"kubernetes.io/projected/d5be666a-6315-45f7-8593-f6474ac6435b-kube-api-access-dwklg\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:00:47.407172 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.407144 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8"] Sep 09 03:00:47.410554 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.410489 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c173ad-68b87cccd7-5xwj8"] Sep 09 03:00:47.437501 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.437463 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:47.565971 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:47.565941 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:48.309945 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:48.309909 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f4d5ed5-0912-4bba-b926-4614ea6a0776" path="/var/lib/kubelet/pods/2f4d5ed5-0912-4bba-b926-4614ea6a0776/volumes" Sep 09 03:00:48.310331 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:48.310279 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5be666a-6315-45f7-8593-f6474ac6435b" path="/var/lib/kubelet/pods/d5be666a-6315-45f7-8593-f6474ac6435b/volumes" Sep 09 03:00:53.309938 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:53.309898 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:53.411835 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:53.411796 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:00:53.555753 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:00:53.555705 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:01:00.687998 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:00.687963 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7849498579-wsxch"] Sep 09 03:01:00.688507 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:00.688206 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7849498579-wsxch" podUID="98f3f0ad-ca4f-4629-bfc0-2ab6aa722003" containerName="manager" containerID="cri-o://b3f58ed68d21bf1d6db46f80b39c2af1793cc8dd6e8d3d7ae03acdfbebb8a01c" gracePeriod=30 Sep 09 03:01:00.938231 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:00.938171 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7849498579-wsxch" Sep 09 03:01:00.981196 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:00.981160 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98f3f0ad-ca4f-4629-bfc0-2ab6aa722003-metrics-certs\") pod \"98f3f0ad-ca4f-4629-bfc0-2ab6aa722003\" (UID: \"98f3f0ad-ca4f-4629-bfc0-2ab6aa722003\") " Sep 09 03:01:00.981399 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:00.981228 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-859s8\" (UniqueName: \"kubernetes.io/projected/98f3f0ad-ca4f-4629-bfc0-2ab6aa722003-kube-api-access-859s8\") pod \"98f3f0ad-ca4f-4629-bfc0-2ab6aa722003\" (UID: \"98f3f0ad-ca4f-4629-bfc0-2ab6aa722003\") " Sep 09 03:01:00.981399 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:00.981315 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/98f3f0ad-ca4f-4629-bfc0-2ab6aa722003-webhook-cert\") pod \"98f3f0ad-ca4f-4629-bfc0-2ab6aa722003\" (UID: \"98f3f0ad-ca4f-4629-bfc0-2ab6aa722003\") " Sep 09 03:01:00.983515 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:00.983470 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f3f0ad-ca4f-4629-bfc0-2ab6aa722003-kube-api-access-859s8" (OuterVolumeSpecName: "kube-api-access-859s8") pod "98f3f0ad-ca4f-4629-bfc0-2ab6aa722003" (UID: "98f3f0ad-ca4f-4629-bfc0-2ab6aa722003"). InnerVolumeSpecName "kube-api-access-859s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 03:01:00.983805 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:00.983780 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98f3f0ad-ca4f-4629-bfc0-2ab6aa722003-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "98f3f0ad-ca4f-4629-bfc0-2ab6aa722003" (UID: "98f3f0ad-ca4f-4629-bfc0-2ab6aa722003"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 03:01:00.983920 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:00.983878 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98f3f0ad-ca4f-4629-bfc0-2ab6aa722003-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "98f3f0ad-ca4f-4629-bfc0-2ab6aa722003" (UID: "98f3f0ad-ca4f-4629-bfc0-2ab6aa722003"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 03:01:01.082488 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:01.082443 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-859s8\" (UniqueName: \"kubernetes.io/projected/98f3f0ad-ca4f-4629-bfc0-2ab6aa722003-kube-api-access-859s8\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:01:01.082488 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:01.082482 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/98f3f0ad-ca4f-4629-bfc0-2ab6aa722003-webhook-cert\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:01:01.082488 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:01.082494 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98f3f0ad-ca4f-4629-bfc0-2ab6aa722003-metrics-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:01:01.159658 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:01.159626 2577 generic.go:356] "Generic (PLEG): container finished" podID="98f3f0ad-ca4f-4629-bfc0-2ab6aa722003" containerID="b3f58ed68d21bf1d6db46f80b39c2af1793cc8dd6e8d3d7ae03acdfbebb8a01c" exitCode=0 Sep 09 03:01:01.159874 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:01.159679 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7849498579-wsxch" Sep 09 03:01:01.159874 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:01.159691 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7849498579-wsxch" event={"ID":"98f3f0ad-ca4f-4629-bfc0-2ab6aa722003","Type":"ContainerDied","Data":"b3f58ed68d21bf1d6db46f80b39c2af1793cc8dd6e8d3d7ae03acdfbebb8a01c"} Sep 09 03:01:01.159874 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:01.159734 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7849498579-wsxch" event={"ID":"98f3f0ad-ca4f-4629-bfc0-2ab6aa722003","Type":"ContainerDied","Data":"66792ff07a8b8a71a7193ea4dd05ca0d1022fe6f5affb26fed77f52911db49e7"} Sep 09 03:01:01.159874 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:01.159756 2577 scope.go:117] "RemoveContainer" containerID="b3f58ed68d21bf1d6db46f80b39c2af1793cc8dd6e8d3d7ae03acdfbebb8a01c" Sep 09 03:01:01.169993 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:01.169973 2577 scope.go:117] "RemoveContainer" containerID="b3f58ed68d21bf1d6db46f80b39c2af1793cc8dd6e8d3d7ae03acdfbebb8a01c" Sep 09 03:01:01.170278 ip-10-0-137-37 kubenswrapper[2577]: E0909 03:01:01.170260 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3f58ed68d21bf1d6db46f80b39c2af1793cc8dd6e8d3d7ae03acdfbebb8a01c\": container with ID starting with b3f58ed68d21bf1d6db46f80b39c2af1793cc8dd6e8d3d7ae03acdfbebb8a01c not found: ID does not exist" containerID="b3f58ed68d21bf1d6db46f80b39c2af1793cc8dd6e8d3d7ae03acdfbebb8a01c" Sep 09 03:01:01.170352 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:01.170285 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f58ed68d21bf1d6db46f80b39c2af1793cc8dd6e8d3d7ae03acdfbebb8a01c"} err="failed to get container status \"b3f58ed68d21bf1d6db46f80b39c2af1793cc8dd6e8d3d7ae03acdfbebb8a01c\": rpc error: code = NotFound desc = could not find container \"b3f58ed68d21bf1d6db46f80b39c2af1793cc8dd6e8d3d7ae03acdfbebb8a01c\": container with ID starting with b3f58ed68d21bf1d6db46f80b39c2af1793cc8dd6e8d3d7ae03acdfbebb8a01c not found: ID does not exist" Sep 09 03:01:01.180245 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:01.180220 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7849498579-wsxch"] Sep 09 03:01:01.183742 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:01.183721 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7849498579-wsxch"] Sep 09 03:01:02.310188 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:02.310155 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98f3f0ad-ca4f-4629-bfc0-2ab6aa722003" path="/var/lib/kubelet/pods/98f3f0ad-ca4f-4629-bfc0-2ab6aa722003/volumes" Sep 09 03:01:57.227902 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:01:57.227866 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:02:02.051561 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.051518 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7849498579-2sm56"] Sep 09 03:02:02.052073 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052011 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18778870-20d2-4e85-b4c8-52dbd1933b24" containerName="maas-api" Sep 09 03:02:02.052073 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052026 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="18778870-20d2-4e85-b4c8-52dbd1933b24" containerName="maas-api" Sep 09 03:02:02.052073 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052041 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4aeb107-020b-4588-943d-ce3f93f8d571" containerName="istio-proxy" Sep 09 03:02:02.052073 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052047 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4aeb107-020b-4588-943d-ce3f93f8d571" containerName="istio-proxy" Sep 09 03:02:02.052073 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052059 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5be666a-6315-45f7-8593-f6474ac6435b" containerName="payload-processing" Sep 09 03:02:02.052073 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052065 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5be666a-6315-45f7-8593-f6474ac6435b" containerName="payload-processing" Sep 09 03:02:02.052073 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052074 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0bf842c-206e-40ca-9d32-602d6d3fcd9b" containerName="maas-api" Sep 09 03:02:02.052073 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052081 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0bf842c-206e-40ca-9d32-602d6d3fcd9b" containerName="maas-api" Sep 09 03:02:02.052443 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052089 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f4d5ed5-0912-4bba-b926-4614ea6a0776" containerName="maas-api" Sep 09 03:02:02.052443 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052094 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f4d5ed5-0912-4bba-b926-4614ea6a0776" containerName="maas-api" Sep 09 03:02:02.052443 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052099 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="977c0787-17ba-4277-902c-e0539fd4c719" containerName="maas-api" Sep 09 03:02:02.052443 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052104 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="977c0787-17ba-4277-902c-e0539fd4c719" containerName="maas-api" Sep 09 03:02:02.052443 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052113 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38c3ad74-d112-4a2f-a7e1-8fc14b02853b" containerName="maas-api" Sep 09 03:02:02.052443 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052117 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="38c3ad74-d112-4a2f-a7e1-8fc14b02853b" containerName="maas-api" Sep 09 03:02:02.052443 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052124 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f3f0ad-ca4f-4629-bfc0-2ab6aa722003" containerName="manager" Sep 09 03:02:02.052443 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052129 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f3f0ad-ca4f-4629-bfc0-2ab6aa722003" containerName="manager" Sep 09 03:02:02.052443 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052187 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="98f3f0ad-ca4f-4629-bfc0-2ab6aa722003" containerName="manager" Sep 09 03:02:02.052443 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052196 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="977c0787-17ba-4277-902c-e0539fd4c719" containerName="maas-api" Sep 09 03:02:02.052443 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052202 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="38c3ad74-d112-4a2f-a7e1-8fc14b02853b" containerName="maas-api" Sep 09 03:02:02.052443 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052208 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="18778870-20d2-4e85-b4c8-52dbd1933b24" containerName="maas-api" Sep 09 03:02:02.052443 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052214 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f4d5ed5-0912-4bba-b926-4614ea6a0776" containerName="maas-api" Sep 09 03:02:02.052443 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052221 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4aeb107-020b-4588-943d-ce3f93f8d571" containerName="istio-proxy" Sep 09 03:02:02.052443 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052225 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5be666a-6315-45f7-8593-f6474ac6435b" containerName="payload-processing" Sep 09 03:02:02.052443 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.052231 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0bf842c-206e-40ca-9d32-602d6d3fcd9b" containerName="maas-api" Sep 09 03:02:02.055266 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.055250 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7849498579-2sm56" Sep 09 03:02:02.058834 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.058800 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zlr72\"" Sep 09 03:02:02.058958 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.058840 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 09 03:02:02.058958 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.058801 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 09 03:02:02.058958 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.058895 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 09 03:02:02.064178 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.064155 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7849498579-2sm56"] Sep 09 03:02:02.142478 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.142441 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf82df2d-fe21-44f5-9a08-39f6992f59e2-metrics-certs\") pod \"maas-controller-7849498579-2sm56\" (UID: \"bf82df2d-fe21-44f5-9a08-39f6992f59e2\") " pod="opendatahub/maas-controller-7849498579-2sm56" Sep 09 03:02:02.142704 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.142528 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf82df2d-fe21-44f5-9a08-39f6992f59e2-webhook-cert\") pod \"maas-controller-7849498579-2sm56\" (UID: \"bf82df2d-fe21-44f5-9a08-39f6992f59e2\") " pod="opendatahub/maas-controller-7849498579-2sm56" Sep 09 03:02:02.142704 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.142595 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cwqz\" (UniqueName: \"kubernetes.io/projected/bf82df2d-fe21-44f5-9a08-39f6992f59e2-kube-api-access-9cwqz\") pod \"maas-controller-7849498579-2sm56\" (UID: \"bf82df2d-fe21-44f5-9a08-39f6992f59e2\") " pod="opendatahub/maas-controller-7849498579-2sm56" Sep 09 03:02:02.244066 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.244035 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf82df2d-fe21-44f5-9a08-39f6992f59e2-metrics-certs\") pod \"maas-controller-7849498579-2sm56\" (UID: \"bf82df2d-fe21-44f5-9a08-39f6992f59e2\") " pod="opendatahub/maas-controller-7849498579-2sm56" Sep 09 03:02:02.244217 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.244097 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf82df2d-fe21-44f5-9a08-39f6992f59e2-webhook-cert\") pod \"maas-controller-7849498579-2sm56\" (UID: \"bf82df2d-fe21-44f5-9a08-39f6992f59e2\") " pod="opendatahub/maas-controller-7849498579-2sm56" Sep 09 03:02:02.244217 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.244143 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cwqz\" (UniqueName: \"kubernetes.io/projected/bf82df2d-fe21-44f5-9a08-39f6992f59e2-kube-api-access-9cwqz\") pod \"maas-controller-7849498579-2sm56\" (UID: \"bf82df2d-fe21-44f5-9a08-39f6992f59e2\") " pod="opendatahub/maas-controller-7849498579-2sm56" Sep 09 03:02:02.246556 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.246532 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf82df2d-fe21-44f5-9a08-39f6992f59e2-metrics-certs\") pod \"maas-controller-7849498579-2sm56\" (UID: \"bf82df2d-fe21-44f5-9a08-39f6992f59e2\") " pod="opendatahub/maas-controller-7849498579-2sm56" Sep 09 03:02:02.246691 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.246600 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf82df2d-fe21-44f5-9a08-39f6992f59e2-webhook-cert\") pod \"maas-controller-7849498579-2sm56\" (UID: \"bf82df2d-fe21-44f5-9a08-39f6992f59e2\") " pod="opendatahub/maas-controller-7849498579-2sm56" Sep 09 03:02:02.252303 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.252274 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cwqz\" (UniqueName: \"kubernetes.io/projected/bf82df2d-fe21-44f5-9a08-39f6992f59e2-kube-api-access-9cwqz\") pod \"maas-controller-7849498579-2sm56\" (UID: \"bf82df2d-fe21-44f5-9a08-39f6992f59e2\") " pod="opendatahub/maas-controller-7849498579-2sm56" Sep 09 03:02:02.367008 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.366914 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7849498579-2sm56" Sep 09 03:02:02.639132 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.639109 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7849498579-2sm56"] Sep 09 03:02:02.641480 ip-10-0-137-37 kubenswrapper[2577]: W0909 03:02:02.641454 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf82df2d_fe21_44f5_9a08_39f6992f59e2.slice/crio-698f437124706494b11fa5cc977ae53a754a4899fca5cd9496c4c7c733e8b413 WatchSource:0}: Error finding container 698f437124706494b11fa5cc977ae53a754a4899fca5cd9496c4c7c733e8b413: Status 404 returned error can't find the container with id 698f437124706494b11fa5cc977ae53a754a4899fca5cd9496c4c7c733e8b413 Sep 09 03:02:02.643408 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:02.643386 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 03:02:03.463013 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:03.462982 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7849498579-2sm56" event={"ID":"bf82df2d-fe21-44f5-9a08-39f6992f59e2","Type":"ContainerStarted","Data":"8a81fe2e51374ab902919b9048daa5d080edca8cfa1aca4c5a7d2871ac0009f5"} Sep 09 03:02:03.463497 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:03.463034 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7849498579-2sm56" event={"ID":"bf82df2d-fe21-44f5-9a08-39f6992f59e2","Type":"ContainerStarted","Data":"698f437124706494b11fa5cc977ae53a754a4899fca5cd9496c4c7c733e8b413"} Sep 09 03:02:03.463497 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:03.463069 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7849498579-2sm56" Sep 09 03:02:03.493265 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:03.493155 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7849498579-2sm56" podStartSLOduration=0.900040308 podStartE2EDuration="1.493135552s" totalImagesPullingTime="593.095244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:02:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:02:03.491202487 +0000 UTC m=+1911.621444582" watchObservedRunningTime="2026-09-09 03:02:03.493135552 +0000 UTC m=+1911.623377692" Sep 09 03:02:12.687177 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:12.687142 2577 scope.go:117] "RemoveContainer" containerID="001f17052ff6444be68c3279750916e65afa6399ba4e85f4a6e73a93e8beb2b1" Sep 09 03:02:12.697372 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:12.697347 2577 scope.go:117] "RemoveContainer" containerID="a620c465e2e543f9acd8999624f57616cb87bd125fee481a6b61146690eaec52" Sep 09 03:02:14.470301 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:14.470269 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7849498579-2sm56" Sep 09 03:02:22.480609 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:22.480564 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:02:23.759768 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:23.759735 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:02:23.873371 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:23.873338 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7849498579-2sm56"] Sep 09 03:02:23.873568 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:23.873537 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7849498579-2sm56" podUID="bf82df2d-fe21-44f5-9a08-39f6992f59e2" containerName="manager" containerID="cri-o://8a81fe2e51374ab902919b9048daa5d080edca8cfa1aca4c5a7d2871ac0009f5" gracePeriod=30 Sep 09 03:02:24.121297 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:24.121275 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7849498579-2sm56" Sep 09 03:02:24.244773 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:24.244740 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf82df2d-fe21-44f5-9a08-39f6992f59e2-metrics-certs\") pod \"bf82df2d-fe21-44f5-9a08-39f6992f59e2\" (UID: \"bf82df2d-fe21-44f5-9a08-39f6992f59e2\") " Sep 09 03:02:24.244980 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:24.244849 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cwqz\" (UniqueName: \"kubernetes.io/projected/bf82df2d-fe21-44f5-9a08-39f6992f59e2-kube-api-access-9cwqz\") pod \"bf82df2d-fe21-44f5-9a08-39f6992f59e2\" (UID: \"bf82df2d-fe21-44f5-9a08-39f6992f59e2\") " Sep 09 03:02:24.244980 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:24.244933 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf82df2d-fe21-44f5-9a08-39f6992f59e2-webhook-cert\") pod \"bf82df2d-fe21-44f5-9a08-39f6992f59e2\" (UID: \"bf82df2d-fe21-44f5-9a08-39f6992f59e2\") " Sep 09 03:02:24.246805 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:24.246775 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf82df2d-fe21-44f5-9a08-39f6992f59e2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bf82df2d-fe21-44f5-9a08-39f6992f59e2" (UID: "bf82df2d-fe21-44f5-9a08-39f6992f59e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 03:02:24.246957 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:24.246903 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf82df2d-fe21-44f5-9a08-39f6992f59e2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "bf82df2d-fe21-44f5-9a08-39f6992f59e2" (UID: "bf82df2d-fe21-44f5-9a08-39f6992f59e2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 03:02:24.246957 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:24.246903 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf82df2d-fe21-44f5-9a08-39f6992f59e2-kube-api-access-9cwqz" (OuterVolumeSpecName: "kube-api-access-9cwqz") pod "bf82df2d-fe21-44f5-9a08-39f6992f59e2" (UID: "bf82df2d-fe21-44f5-9a08-39f6992f59e2"). InnerVolumeSpecName "kube-api-access-9cwqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 03:02:24.345956 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:24.345930 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf82df2d-fe21-44f5-9a08-39f6992f59e2-metrics-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:02:24.345956 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:24.345952 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cwqz\" (UniqueName: \"kubernetes.io/projected/bf82df2d-fe21-44f5-9a08-39f6992f59e2-kube-api-access-9cwqz\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:02:24.346098 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:24.345966 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf82df2d-fe21-44f5-9a08-39f6992f59e2-webhook-cert\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:02:24.565992 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:24.565957 2577 generic.go:356] "Generic (PLEG): container finished" podID="bf82df2d-fe21-44f5-9a08-39f6992f59e2" containerID="8a81fe2e51374ab902919b9048daa5d080edca8cfa1aca4c5a7d2871ac0009f5" exitCode=0 Sep 09 03:02:24.566173 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:24.566007 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7849498579-2sm56" Sep 09 03:02:24.566173 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:24.566016 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7849498579-2sm56" event={"ID":"bf82df2d-fe21-44f5-9a08-39f6992f59e2","Type":"ContainerDied","Data":"8a81fe2e51374ab902919b9048daa5d080edca8cfa1aca4c5a7d2871ac0009f5"} Sep 09 03:02:24.566173 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:24.566071 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7849498579-2sm56" event={"ID":"bf82df2d-fe21-44f5-9a08-39f6992f59e2","Type":"ContainerDied","Data":"698f437124706494b11fa5cc977ae53a754a4899fca5cd9496c4c7c733e8b413"} Sep 09 03:02:24.566173 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:24.566103 2577 scope.go:117] "RemoveContainer" containerID="8a81fe2e51374ab902919b9048daa5d080edca8cfa1aca4c5a7d2871ac0009f5" Sep 09 03:02:24.574729 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:24.574705 2577 scope.go:117] "RemoveContainer" containerID="8a81fe2e51374ab902919b9048daa5d080edca8cfa1aca4c5a7d2871ac0009f5" Sep 09 03:02:24.575030 ip-10-0-137-37 kubenswrapper[2577]: E0909 03:02:24.575010 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a81fe2e51374ab902919b9048daa5d080edca8cfa1aca4c5a7d2871ac0009f5\": container with ID starting with 8a81fe2e51374ab902919b9048daa5d080edca8cfa1aca4c5a7d2871ac0009f5 not found: ID does not exist" containerID="8a81fe2e51374ab902919b9048daa5d080edca8cfa1aca4c5a7d2871ac0009f5" Sep 09 03:02:24.575109 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:24.575037 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a81fe2e51374ab902919b9048daa5d080edca8cfa1aca4c5a7d2871ac0009f5"} err="failed to get container status \"8a81fe2e51374ab902919b9048daa5d080edca8cfa1aca4c5a7d2871ac0009f5\": rpc error: code = NotFound desc = could not find container \"8a81fe2e51374ab902919b9048daa5d080edca8cfa1aca4c5a7d2871ac0009f5\": container with ID starting with 8a81fe2e51374ab902919b9048daa5d080edca8cfa1aca4c5a7d2871ac0009f5 not found: ID does not exist" Sep 09 03:02:24.580990 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:24.580970 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7849498579-2sm56"] Sep 09 03:02:24.584520 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:24.584498 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7849498579-2sm56"] Sep 09 03:02:25.260129 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.260087 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:02:25.362855 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.362811 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7849498579-zgmkn"] Sep 09 03:02:25.363254 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.363231 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4aeb107-020b-4588-943d-ce3f93f8d571" containerName="istio-proxy" Sep 09 03:02:25.363254 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.363250 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4aeb107-020b-4588-943d-ce3f93f8d571" containerName="istio-proxy" Sep 09 03:02:25.363418 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.363262 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf82df2d-fe21-44f5-9a08-39f6992f59e2" containerName="manager" Sep 09 03:02:25.363418 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.363269 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf82df2d-fe21-44f5-9a08-39f6992f59e2" containerName="manager" Sep 09 03:02:25.363418 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.363333 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf82df2d-fe21-44f5-9a08-39f6992f59e2" containerName="manager" Sep 09 03:02:25.363418 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.363345 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4aeb107-020b-4588-943d-ce3f93f8d571" containerName="istio-proxy" Sep 09 03:02:25.366138 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.366118 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7849498579-zgmkn" Sep 09 03:02:25.369756 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.369736 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zlr72\"" Sep 09 03:02:25.369903 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.369771 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 09 03:02:25.369903 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.369800 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 09 03:02:25.370020 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.370007 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 09 03:02:25.375428 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.375409 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7849498579-zgmkn"] Sep 09 03:02:25.455268 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.455240 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc9wq\" (UniqueName: \"kubernetes.io/projected/5fd71c8a-79ad-4462-90a2-a75283e40cb2-kube-api-access-qc9wq\") pod \"maas-controller-7849498579-zgmkn\" (UID: \"5fd71c8a-79ad-4462-90a2-a75283e40cb2\") " pod="opendatahub/maas-controller-7849498579-zgmkn" Sep 09 03:02:25.455437 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.455297 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5fd71c8a-79ad-4462-90a2-a75283e40cb2-webhook-cert\") pod \"maas-controller-7849498579-zgmkn\" (UID: \"5fd71c8a-79ad-4462-90a2-a75283e40cb2\") " pod="opendatahub/maas-controller-7849498579-zgmkn" Sep 09 03:02:25.455437 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.455331 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fd71c8a-79ad-4462-90a2-a75283e40cb2-metrics-certs\") pod \"maas-controller-7849498579-zgmkn\" (UID: \"5fd71c8a-79ad-4462-90a2-a75283e40cb2\") " pod="opendatahub/maas-controller-7849498579-zgmkn" Sep 09 03:02:25.556391 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.556315 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5fd71c8a-79ad-4462-90a2-a75283e40cb2-webhook-cert\") pod \"maas-controller-7849498579-zgmkn\" (UID: \"5fd71c8a-79ad-4462-90a2-a75283e40cb2\") " pod="opendatahub/maas-controller-7849498579-zgmkn" Sep 09 03:02:25.556391 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.556358 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fd71c8a-79ad-4462-90a2-a75283e40cb2-metrics-certs\") pod \"maas-controller-7849498579-zgmkn\" (UID: \"5fd71c8a-79ad-4462-90a2-a75283e40cb2\") " pod="opendatahub/maas-controller-7849498579-zgmkn" Sep 09 03:02:25.556590 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.556421 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc9wq\" (UniqueName: \"kubernetes.io/projected/5fd71c8a-79ad-4462-90a2-a75283e40cb2-kube-api-access-qc9wq\") pod \"maas-controller-7849498579-zgmkn\" (UID: \"5fd71c8a-79ad-4462-90a2-a75283e40cb2\") " pod="opendatahub/maas-controller-7849498579-zgmkn" Sep 09 03:02:25.558743 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.558715 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fd71c8a-79ad-4462-90a2-a75283e40cb2-metrics-certs\") pod \"maas-controller-7849498579-zgmkn\" (UID: \"5fd71c8a-79ad-4462-90a2-a75283e40cb2\") " pod="opendatahub/maas-controller-7849498579-zgmkn" Sep 09 03:02:25.558885 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.558777 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5fd71c8a-79ad-4462-90a2-a75283e40cb2-webhook-cert\") pod \"maas-controller-7849498579-zgmkn\" (UID: \"5fd71c8a-79ad-4462-90a2-a75283e40cb2\") " pod="opendatahub/maas-controller-7849498579-zgmkn" Sep 09 03:02:25.563585 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.563561 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc9wq\" (UniqueName: \"kubernetes.io/projected/5fd71c8a-79ad-4462-90a2-a75283e40cb2-kube-api-access-qc9wq\") pod \"maas-controller-7849498579-zgmkn\" (UID: \"5fd71c8a-79ad-4462-90a2-a75283e40cb2\") " pod="opendatahub/maas-controller-7849498579-zgmkn" Sep 09 03:02:25.676652 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.676620 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7849498579-zgmkn" Sep 09 03:02:25.946673 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:25.946646 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7849498579-zgmkn"] Sep 09 03:02:25.948640 ip-10-0-137-37 kubenswrapper[2577]: W0909 03:02:25.948606 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd71c8a_79ad_4462_90a2_a75283e40cb2.slice/crio-f29bfbc860986ed4be576f704b7abc1c44647b4d197c272bbfc949d8c85f4af4 WatchSource:0}: Error finding container f29bfbc860986ed4be576f704b7abc1c44647b4d197c272bbfc949d8c85f4af4: Status 404 returned error can't find the container with id f29bfbc860986ed4be576f704b7abc1c44647b4d197c272bbfc949d8c85f4af4 Sep 09 03:02:26.309855 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:26.309758 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf82df2d-fe21-44f5-9a08-39f6992f59e2" path="/var/lib/kubelet/pods/bf82df2d-fe21-44f5-9a08-39f6992f59e2/volumes" Sep 09 03:02:26.578556 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:26.578480 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7849498579-zgmkn" event={"ID":"5fd71c8a-79ad-4462-90a2-a75283e40cb2","Type":"ContainerStarted","Data":"77b51053638d5f8b29611b1b9fa0d7982c1569466d8fb0c42c74f0e2f1509218"} Sep 09 03:02:26.578556 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:26.578540 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7849498579-zgmkn" event={"ID":"5fd71c8a-79ad-4462-90a2-a75283e40cb2","Type":"ContainerStarted","Data":"f29bfbc860986ed4be576f704b7abc1c44647b4d197c272bbfc949d8c85f4af4"} Sep 09 03:02:26.578750 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:26.578581 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7849498579-zgmkn" Sep 09 03:02:26.596536 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:26.596489 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7849498579-zgmkn" podStartSLOduration=1.099574714 podStartE2EDuration="1.59647292s" totalImagesPullingTime="496.898206ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:02:26.595314185 +0000 UTC m=+1934.725556282" watchObservedRunningTime="2026-09-09 03:02:26.59647292 +0000 UTC m=+1934.726715015" Sep 09 03:02:37.586864 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:37.586809 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7849498579-zgmkn" Sep 09 03:02:39.155417 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:39.155378 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:02:41.229463 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:02:41.229430 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:03:17.242266 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:03:17.242217 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:04:29.295976 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:04:29.295936 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j529m"] Sep 09 03:04:29.299359 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:04:29.299342 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j529m" Sep 09 03:04:29.301764 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:04:29.301731 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bh6dl\"" Sep 09 03:04:29.301764 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:04:29.301733 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 09 03:04:29.301997 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:04:29.301765 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 09 03:04:29.315537 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:04:29.315513 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j529m"] Sep 09 03:04:29.357652 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:04:29.357614 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fc56e89-472e-48ae-9810-b61b7ff4daa3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j529m\" (UID: \"6fc56e89-472e-48ae-9810-b61b7ff4daa3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j529m" Sep 09 03:04:29.357652 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:04:29.357652 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xkjz\" (UniqueName: \"kubernetes.io/projected/6fc56e89-472e-48ae-9810-b61b7ff4daa3-kube-api-access-4xkjz\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j529m\" (UID: \"6fc56e89-472e-48ae-9810-b61b7ff4daa3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j529m" Sep 09 03:04:29.458434 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:04:29.458395 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xkjz\" (UniqueName: \"kubernetes.io/projected/6fc56e89-472e-48ae-9810-b61b7ff4daa3-kube-api-access-4xkjz\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j529m\" (UID: \"6fc56e89-472e-48ae-9810-b61b7ff4daa3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j529m" Sep 09 03:04:29.458647 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:04:29.458530 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fc56e89-472e-48ae-9810-b61b7ff4daa3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j529m\" (UID: \"6fc56e89-472e-48ae-9810-b61b7ff4daa3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j529m" Sep 09 03:04:29.458951 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:04:29.458931 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fc56e89-472e-48ae-9810-b61b7ff4daa3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j529m\" (UID: \"6fc56e89-472e-48ae-9810-b61b7ff4daa3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j529m" Sep 09 03:04:29.466217 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:04:29.466194 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xkjz\" (UniqueName: \"kubernetes.io/projected/6fc56e89-472e-48ae-9810-b61b7ff4daa3-kube-api-access-4xkjz\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j529m\" (UID: \"6fc56e89-472e-48ae-9810-b61b7ff4daa3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j529m" Sep 09 03:04:29.610432 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:04:29.610343 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j529m" Sep 09 03:04:29.890372 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:04:29.890348 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j529m"] Sep 09 03:04:29.892116 ip-10-0-137-37 kubenswrapper[2577]: W0909 03:04:29.892093 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc56e89_472e_48ae_9810_b61b7ff4daa3.slice/crio-8138b9db62d3b7ec2a2f46cf6d0743d55e91a4652b813139d7ff2f2af6931617 WatchSource:0}: Error finding container 8138b9db62d3b7ec2a2f46cf6d0743d55e91a4652b813139d7ff2f2af6931617: Status 404 returned error can't find the container with id 8138b9db62d3b7ec2a2f46cf6d0743d55e91a4652b813139d7ff2f2af6931617 Sep 09 03:04:30.186179 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:04:30.186090 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j529m" event={"ID":"6fc56e89-472e-48ae-9810-b61b7ff4daa3","Type":"ContainerStarted","Data":"d95b06be5ead7aee7c5fba6f1875b52909b8f1a25787ebd2eeb70b70f7d7bbb0"} Sep 09 03:04:30.186179 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:04:30.186142 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j529m" event={"ID":"6fc56e89-472e-48ae-9810-b61b7ff4daa3","Type":"ContainerStarted","Data":"8138b9db62d3b7ec2a2f46cf6d0743d55e91a4652b813139d7ff2f2af6931617"} Sep 09 03:04:30.186179 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:04:30.186168 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j529m" Sep 09 03:04:30.209180 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:04:30.209129 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j529m" podStartSLOduration=1.209112349 podStartE2EDuration="1.209112349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:04:30.207083206 +0000 UTC m=+2058.337325312" watchObservedRunningTime="2026-09-09 03:04:30.209112349 +0000 UTC m=+2058.339354444" Sep 09 03:04:41.193450 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:04:41.193373 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j529m" Sep 09 03:05:10.602519 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:05:10.602482 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:05:12.487081 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:05:12.487055 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/2.log" Sep 09 03:05:12.488182 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:05:12.488161 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/2.log" Sep 09 03:05:12.739240 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:05:12.739155 2577 scope.go:117] "RemoveContainer" containerID="4e39e4aef300a29a058be9c8c8f949c32b81f5f966ca3eeaace14eac30322487" Sep 09 03:05:20.277060 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:05:20.277018 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:05:43.330419 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:05:43.330378 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:05:44.141738 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:05:44.141703 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:05:46.472753 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:05:46.472720 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:06:02.356755 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:02.356718 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6f6cdc5d86-f45rf"] Sep 09 03:06:02.360760 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:02.360738 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-f45rf" Sep 09 03:06:02.372181 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:02.372156 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f6cdc5d86-f45rf"] Sep 09 03:06:02.402205 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:02.402170 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2602fa88-f4bc-40d7-8a04-068d79ce590d-metrics-certs\") pod \"maas-api-6f6cdc5d86-f45rf\" (UID: \"2602fa88-f4bc-40d7-8a04-068d79ce590d\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-f45rf" Sep 09 03:06:02.402401 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:02.402212 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2602fa88-f4bc-40d7-8a04-068d79ce590d-maas-api-tls\") pod \"maas-api-6f6cdc5d86-f45rf\" (UID: \"2602fa88-f4bc-40d7-8a04-068d79ce590d\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-f45rf" Sep 09 03:06:02.402401 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:02.402258 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2602fa88-f4bc-40d7-8a04-068d79ce590d-openshift-service-ca\") pod \"maas-api-6f6cdc5d86-f45rf\" (UID: \"2602fa88-f4bc-40d7-8a04-068d79ce590d\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-f45rf" Sep 09 03:06:02.402401 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:02.402334 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm98s\" (UniqueName: \"kubernetes.io/projected/2602fa88-f4bc-40d7-8a04-068d79ce590d-kube-api-access-lm98s\") pod \"maas-api-6f6cdc5d86-f45rf\" (UID: \"2602fa88-f4bc-40d7-8a04-068d79ce590d\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-f45rf" Sep 09 03:06:02.503592 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:02.503549 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2602fa88-f4bc-40d7-8a04-068d79ce590d-metrics-certs\") pod \"maas-api-6f6cdc5d86-f45rf\" (UID: \"2602fa88-f4bc-40d7-8a04-068d79ce590d\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-f45rf" Sep 09 03:06:02.503592 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:02.503594 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2602fa88-f4bc-40d7-8a04-068d79ce590d-maas-api-tls\") pod \"maas-api-6f6cdc5d86-f45rf\" (UID: \"2602fa88-f4bc-40d7-8a04-068d79ce590d\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-f45rf" Sep 09 03:06:02.503913 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:02.503633 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2602fa88-f4bc-40d7-8a04-068d79ce590d-openshift-service-ca\") pod \"maas-api-6f6cdc5d86-f45rf\" (UID: \"2602fa88-f4bc-40d7-8a04-068d79ce590d\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-f45rf" Sep 09 03:06:02.503913 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:02.503664 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm98s\" (UniqueName: \"kubernetes.io/projected/2602fa88-f4bc-40d7-8a04-068d79ce590d-kube-api-access-lm98s\") pod \"maas-api-6f6cdc5d86-f45rf\" (UID: \"2602fa88-f4bc-40d7-8a04-068d79ce590d\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-f45rf" Sep 09 03:06:02.504409 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:02.504388 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2602fa88-f4bc-40d7-8a04-068d79ce590d-openshift-service-ca\") pod \"maas-api-6f6cdc5d86-f45rf\" (UID: \"2602fa88-f4bc-40d7-8a04-068d79ce590d\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-f45rf" Sep 09 03:06:02.506255 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:02.506234 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2602fa88-f4bc-40d7-8a04-068d79ce590d-maas-api-tls\") pod \"maas-api-6f6cdc5d86-f45rf\" (UID: \"2602fa88-f4bc-40d7-8a04-068d79ce590d\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-f45rf" Sep 09 03:06:02.506362 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:02.506283 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2602fa88-f4bc-40d7-8a04-068d79ce590d-metrics-certs\") pod \"maas-api-6f6cdc5d86-f45rf\" (UID: \"2602fa88-f4bc-40d7-8a04-068d79ce590d\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-f45rf" Sep 09 03:06:02.511203 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:02.511178 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm98s\" (UniqueName: \"kubernetes.io/projected/2602fa88-f4bc-40d7-8a04-068d79ce590d-kube-api-access-lm98s\") pod \"maas-api-6f6cdc5d86-f45rf\" (UID: \"2602fa88-f4bc-40d7-8a04-068d79ce590d\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-f45rf" Sep 09 03:06:02.672931 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:02.672900 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-f45rf" Sep 09 03:06:02.945105 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:02.945057 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f6cdc5d86-f45rf"] Sep 09 03:06:02.947977 ip-10-0-137-37 kubenswrapper[2577]: W0909 03:06:02.947949 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2602fa88_f4bc_40d7_8a04_068d79ce590d.slice/crio-73fc335483f3bf27623a0e5a50ba9203c366c990ffc9058841e3cb977d077d02 WatchSource:0}: Error finding container 73fc335483f3bf27623a0e5a50ba9203c366c990ffc9058841e3cb977d077d02: Status 404 returned error can't find the container with id 73fc335483f3bf27623a0e5a50ba9203c366c990ffc9058841e3cb977d077d02 Sep 09 03:06:03.657126 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:03.657092 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-f45rf" event={"ID":"2602fa88-f4bc-40d7-8a04-068d79ce590d","Type":"ContainerStarted","Data":"450a9337e276ea77b3ba049e3328635fbbeac33fc17fa5652d3fa53460bdd71c"} Sep 09 03:06:03.657520 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:03.657146 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-f45rf" event={"ID":"2602fa88-f4bc-40d7-8a04-068d79ce590d","Type":"ContainerStarted","Data":"73fc335483f3bf27623a0e5a50ba9203c366c990ffc9058841e3cb977d077d02"} Sep 09 03:06:03.657520 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:03.657220 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-f45rf" Sep 09 03:06:03.676111 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:03.676059 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-f45rf" podStartSLOduration=1.131445426 podStartE2EDuration="1.676043026s" totalImagesPullingTime="544.5976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:06:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:03.67362567 +0000 UTC m=+2151.803867768" watchObservedRunningTime="2026-09-09 03:06:03.676043026 +0000 UTC m=+2151.806285123" Sep 09 03:06:09.668724 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:09.668649 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-f45rf" Sep 09 03:06:17.799874 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:17.799837 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55c9b5658-6wg48"] Sep 09 03:06:17.800732 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:17.800684 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" podUID="8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4" containerName="maas-api" containerID="cri-o://8e2ee62d33c8e89811797994554c72c4675cb4058f2dfb2ac88d85c6602cdc30" gracePeriod=30 Sep 09 03:06:19.058895 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:19.058870 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" Sep 09 03:06:19.162632 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:19.162524 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfns6\" (UniqueName: \"kubernetes.io/projected/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-kube-api-access-nfns6\") pod \"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4\" (UID: \"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4\") " Sep 09 03:06:19.162632 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:19.162597 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-metrics-certs\") pod \"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4\" (UID: \"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4\") " Sep 09 03:06:19.162919 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:19.162650 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-maas-api-tls\") pod \"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4\" (UID: \"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4\") " Sep 09 03:06:19.162919 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:19.162691 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-openshift-service-ca\") pod \"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4\" (UID: \"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4\") " Sep 09 03:06:19.163210 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:19.163179 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4" (UID: "8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 03:06:19.164859 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:19.164808 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4" (UID: "8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 03:06:19.164971 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:19.164891 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-kube-api-access-nfns6" (OuterVolumeSpecName: "kube-api-access-nfns6") pod "8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4" (UID: "8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4"). InnerVolumeSpecName "kube-api-access-nfns6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 03:06:19.164971 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:19.164911 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4" (UID: "8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 03:06:19.263590 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:19.263552 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-openshift-service-ca\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:06:19.263590 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:19.263581 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfns6\" (UniqueName: \"kubernetes.io/projected/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-kube-api-access-nfns6\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:06:19.263590 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:19.263590 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-metrics-certs\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:06:19.263590 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:19.263598 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4-maas-api-tls\") on node \"ip-10-0-137-37.ec2.internal\" DevicePath \"\"" Sep 09 03:06:19.742292 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:19.742256 2577 generic.go:356] "Generic (PLEG): container finished" podID="8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4" containerID="8e2ee62d33c8e89811797994554c72c4675cb4058f2dfb2ac88d85c6602cdc30" exitCode=0 Sep 09 03:06:19.742497 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:19.742307 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" Sep 09 03:06:19.742497 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:19.742341 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" event={"ID":"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4","Type":"ContainerDied","Data":"8e2ee62d33c8e89811797994554c72c4675cb4058f2dfb2ac88d85c6602cdc30"} Sep 09 03:06:19.742497 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:19.742390 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55c9b5658-6wg48" event={"ID":"8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4","Type":"ContainerDied","Data":"7ea8af1b48b02042174bc1fc706a810f22217d2278b9c09e4cd80917c67cee85"} Sep 09 03:06:19.742497 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:19.742419 2577 scope.go:117] "RemoveContainer" containerID="8e2ee62d33c8e89811797994554c72c4675cb4058f2dfb2ac88d85c6602cdc30" Sep 09 03:06:19.751968 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:19.751947 2577 scope.go:117] "RemoveContainer" containerID="8e2ee62d33c8e89811797994554c72c4675cb4058f2dfb2ac88d85c6602cdc30" Sep 09 03:06:19.752477 ip-10-0-137-37 kubenswrapper[2577]: E0909 03:06:19.752301 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2ee62d33c8e89811797994554c72c4675cb4058f2dfb2ac88d85c6602cdc30\": container with ID starting with 8e2ee62d33c8e89811797994554c72c4675cb4058f2dfb2ac88d85c6602cdc30 not found: ID does not exist" containerID="8e2ee62d33c8e89811797994554c72c4675cb4058f2dfb2ac88d85c6602cdc30" Sep 09 03:06:19.752477 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:19.752336 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2ee62d33c8e89811797994554c72c4675cb4058f2dfb2ac88d85c6602cdc30"} err="failed to get container status \"8e2ee62d33c8e89811797994554c72c4675cb4058f2dfb2ac88d85c6602cdc30\": rpc error: code = NotFound desc = could not find container \"8e2ee62d33c8e89811797994554c72c4675cb4058f2dfb2ac88d85c6602cdc30\": container with ID starting with 8e2ee62d33c8e89811797994554c72c4675cb4058f2dfb2ac88d85c6602cdc30 not found: ID does not exist" Sep 09 03:06:19.770282 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:19.770251 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55c9b5658-6wg48"] Sep 09 03:06:19.773889 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:19.773859 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-55c9b5658-6wg48"] Sep 09 03:06:20.310383 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:20.310347 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4" path="/var/lib/kubelet/pods/8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4/volumes" Sep 09 03:06:37.893403 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:37.893362 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7849498579-zgmkn_5fd71c8a-79ad-4462-90a2-a75283e40cb2/manager/0.log" Sep 09 03:06:38.257196 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:38.257119 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dfc788bdd-4gjx2_0377c506-1c88-4580-bcfc-c81bd78e3f30/copy-manifests/0.log" Sep 09 03:06:38.264827 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:38.264784 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dfc788bdd-4gjx2_0377c506-1c88-4580-bcfc-c81bd78e3f30/manager/0.log" Sep 09 03:06:38.858729 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:38.858695 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6f6cdc5d86-f45rf_2602fa88-f4bc-40d7-8a04-068d79ce590d/maas-api/0.log" Sep 09 03:06:39.320774 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:39.320740 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-78hwb_4ed93ff6-dddd-4e46-9eee-850b42de02f3/postgres/0.log" Sep 09 03:06:40.659692 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:40.659664 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-j529m_6fc56e89-472e-48ae-9810-b61b7ff4daa3/manager/0.log" Sep 09 03:06:41.651037 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:41.651014 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7dc95696d9-wz9vg_74f65ba8-8aae-4b5a-9229-0813168e1063/kube-auth-proxy/0.log" Sep 09 03:06:41.899965 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:41.899936 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-rggjg_7bd8d2bf-69f5-4f60-ae12-ff382fa47f7e/payload-pre-processing/0.log" Sep 09 03:06:42.508253 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:42.508218 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-xfjjl_78249e2b-fb45-4687-bf00-22c6e47645ef/main/0.log" Sep 09 03:06:49.006711 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:49.006674 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:06:49.079012 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:49.078975 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" limit=1024 Sep 09 03:06:51.902452 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:51.902407 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4fb84_e0fea0ae-0424-4574-b264-04ea4bcdea35/global-pull-secret-syncer/0.log" Sep 09 03:06:52.062443 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:52.062391 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9jpwt_6022d651-c68e-42a6-86c9-eee48a15f8a2/konnectivity-agent/0.log" Sep 09 03:06:52.199514 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:52.199427 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-37.ec2.internal_60f11ba69470c54c65a13a194197ca4c/haproxy/0.log" Sep 09 03:06:56.885719 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:56.885683 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-j529m_6fc56e89-472e-48ae-9810-b61b7ff4daa3/manager/0.log" Sep 09 03:06:58.422261 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:58.422214 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fa55c8a-6da0-46e5-b559-e13972dee6e7/alertmanager/0.log" Sep 09 03:06:58.444056 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:58.444024 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fa55c8a-6da0-46e5-b559-e13972dee6e7/config-reloader/0.log" Sep 09 03:06:58.467183 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:58.467157 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fa55c8a-6da0-46e5-b559-e13972dee6e7/kube-rbac-proxy-web/0.log" Sep 09 03:06:58.490865 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:58.490839 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fa55c8a-6da0-46e5-b559-e13972dee6e7/kube-rbac-proxy/0.log" Sep 09 03:06:58.515626 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:58.515598 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fa55c8a-6da0-46e5-b559-e13972dee6e7/kube-rbac-proxy-metric/0.log" Sep 09 03:06:58.542229 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:58.542202 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fa55c8a-6da0-46e5-b559-e13972dee6e7/prom-label-proxy/0.log" Sep 09 03:06:58.564237 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:58.564207 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fa55c8a-6da0-46e5-b559-e13972dee6e7/init-config-reloader/0.log" Sep 09 03:06:58.646639 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:58.646549 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-ztr5j_fb28dabe-ca30-4d6c-83bb-271e6751e5e1/kube-state-metrics/0.log" Sep 09 03:06:58.668910 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:58.668887 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-ztr5j_fb28dabe-ca30-4d6c-83bb-271e6751e5e1/kube-rbac-proxy-main/0.log" Sep 09 03:06:58.697261 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:58.697229 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-ztr5j_fb28dabe-ca30-4d6c-83bb-271e6751e5e1/kube-rbac-proxy-self/0.log" Sep 09 03:06:58.809648 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:58.809617 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-g26j9_b01cd589-629e-4438-b7c0-758160094ef9/monitoring-plugin/0.log" Sep 09 03:06:58.846661 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:58.846633 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bbcsl_833e16d1-4060-4d33-aff7-945b2a7c2786/node-exporter/0.log" Sep 09 03:06:58.877928 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:58.877906 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bbcsl_833e16d1-4060-4d33-aff7-945b2a7c2786/kube-rbac-proxy/0.log" Sep 09 03:06:58.921887 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:58.921791 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bbcsl_833e16d1-4060-4d33-aff7-945b2a7c2786/init-textfile/0.log" Sep 09 03:06:59.214752 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:59.214672 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-9h959_654087bd-60b8-4b72-8104-ce90acbb0379/kube-rbac-proxy-main/0.log" Sep 09 03:06:59.243622 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:59.243577 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-9h959_654087bd-60b8-4b72-8104-ce90acbb0379/kube-rbac-proxy-self/0.log" Sep 09 03:06:59.269755 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:59.269724 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-9h959_654087bd-60b8-4b72-8104-ce90acbb0379/openshift-state-metrics/0.log" Sep 09 03:06:59.343349 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:59.343278 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_73dcafda-97f5-4955-9bb2-cdfbf232eae4/prometheus/0.log" Sep 09 03:06:59.376181 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:59.376157 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_73dcafda-97f5-4955-9bb2-cdfbf232eae4/config-reloader/0.log" Sep 09 03:06:59.439971 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:59.439945 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_73dcafda-97f5-4955-9bb2-cdfbf232eae4/thanos-sidecar/0.log" Sep 09 03:06:59.475425 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:59.475349 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_73dcafda-97f5-4955-9bb2-cdfbf232eae4/kube-rbac-proxy-web/0.log" Sep 09 03:06:59.518169 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:59.518140 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_73dcafda-97f5-4955-9bb2-cdfbf232eae4/kube-rbac-proxy/0.log" Sep 09 03:06:59.541994 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:59.541966 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_73dcafda-97f5-4955-9bb2-cdfbf232eae4/kube-rbac-proxy-thanos/0.log" Sep 09 03:06:59.564690 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:59.564670 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_73dcafda-97f5-4955-9bb2-cdfbf232eae4/init-config-reloader/0.log" Sep 09 03:06:59.652939 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:59.652911 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-kjfz6_f6e44699-f59e-45df-a19a-939054986aeb/prometheus-operator-admission-webhook/0.log" Sep 09 03:06:59.687704 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:59.687677 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54b8dcbdd-9zrn6_ea9506c8-63c6-4868-b855-d938d9858e35/telemeter-client/0.log" Sep 09 03:06:59.713639 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:59.713613 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54b8dcbdd-9zrn6_ea9506c8-63c6-4868-b855-d938d9858e35/reload/0.log" Sep 09 03:06:59.736224 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:59.736137 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54b8dcbdd-9zrn6_ea9506c8-63c6-4868-b855-d938d9858e35/kube-rbac-proxy/0.log" Sep 09 03:06:59.777618 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:59.777588 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cb9bb7cb4-mfqnf_15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0/thanos-query/0.log" Sep 09 03:06:59.806254 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:59.806232 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cb9bb7cb4-mfqnf_15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0/kube-rbac-proxy-web/0.log" Sep 09 03:06:59.834516 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:59.834491 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cb9bb7cb4-mfqnf_15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0/kube-rbac-proxy/0.log" Sep 09 03:06:59.860848 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:59.860806 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cb9bb7cb4-mfqnf_15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0/prom-label-proxy/0.log" Sep 09 03:06:59.892536 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:59.892507 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cb9bb7cb4-mfqnf_15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0/kube-rbac-proxy-rules/0.log" Sep 09 03:06:59.918068 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:06:59.918015 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cb9bb7cb4-mfqnf_15592ffc-8bd5-4c4d-a69b-6f2d24e8e6d0/kube-rbac-proxy-metrics/0.log" Sep 09 03:07:01.239370 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:01.239339 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-j5465_3f5283dd-ae41-4c1a-a1d3-84285f55e969/networking-console-plugin/0.log" Sep 09 03:07:01.868243 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:01.868210 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/2.log" Sep 09 03:07:01.876054 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:01.876025 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-g849c_bbed170a-4fc8-4c57-a0dd-320c37ea878c/console-operator/3.log" Sep 09 03:07:01.935066 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:01.935036 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs"] Sep 09 03:07:01.935512 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:01.935496 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4" containerName="maas-api" Sep 09 03:07:01.935567 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:01.935514 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4" containerName="maas-api" Sep 09 03:07:01.935610 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:01.935586 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c42873d-b0e9-43a9-aac3-ebca4ff9e1a4" containerName="maas-api" Sep 09 03:07:01.938717 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:01.938698 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs" Sep 09 03:07:01.941386 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:01.941361 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qh97t\"/\"openshift-service-ca.crt\"" Sep 09 03:07:01.942496 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:01.942476 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qh97t\"/\"kube-root-ca.crt\"" Sep 09 03:07:01.942569 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:01.942547 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qh97t\"/\"default-dockercfg-kzxwx\"" Sep 09 03:07:01.949354 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:01.949329 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs"] Sep 09 03:07:02.060086 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:02.060045 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb150ea7-3521-4012-a358-d4abf3b2e2de-sys\") pod \"perf-node-gather-daemonset-p7hqs\" (UID: \"cb150ea7-3521-4012-a358-d4abf3b2e2de\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs" Sep 09 03:07:02.060086 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:02.060090 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb150ea7-3521-4012-a358-d4abf3b2e2de-lib-modules\") pod \"perf-node-gather-daemonset-p7hqs\" (UID: \"cb150ea7-3521-4012-a358-d4abf3b2e2de\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs" Sep 09 03:07:02.060351 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:02.060123 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb150ea7-3521-4012-a358-d4abf3b2e2de-podres\") pod \"perf-node-gather-daemonset-p7hqs\" (UID: \"cb150ea7-3521-4012-a358-d4abf3b2e2de\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs" Sep 09 03:07:02.060351 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:02.060214 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qg2t\" (UniqueName: \"kubernetes.io/projected/cb150ea7-3521-4012-a358-d4abf3b2e2de-kube-api-access-6qg2t\") pod \"perf-node-gather-daemonset-p7hqs\" (UID: \"cb150ea7-3521-4012-a358-d4abf3b2e2de\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs" Sep 09 03:07:02.060351 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:02.060254 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb150ea7-3521-4012-a358-d4abf3b2e2de-proc\") pod \"perf-node-gather-daemonset-p7hqs\" (UID: \"cb150ea7-3521-4012-a358-d4abf3b2e2de\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs" Sep 09 03:07:02.161309 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:02.161271 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb150ea7-3521-4012-a358-d4abf3b2e2de-proc\") pod \"perf-node-gather-daemonset-p7hqs\" (UID: \"cb150ea7-3521-4012-a358-d4abf3b2e2de\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs" Sep 09 03:07:02.161503 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:02.161331 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb150ea7-3521-4012-a358-d4abf3b2e2de-sys\") pod \"perf-node-gather-daemonset-p7hqs\" (UID: \"cb150ea7-3521-4012-a358-d4abf3b2e2de\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs" Sep 09 03:07:02.161503 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:02.161349 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb150ea7-3521-4012-a358-d4abf3b2e2de-lib-modules\") pod \"perf-node-gather-daemonset-p7hqs\" (UID: \"cb150ea7-3521-4012-a358-d4abf3b2e2de\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs" Sep 09 03:07:02.161503 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:02.161370 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb150ea7-3521-4012-a358-d4abf3b2e2de-podres\") pod \"perf-node-gather-daemonset-p7hqs\" (UID: \"cb150ea7-3521-4012-a358-d4abf3b2e2de\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs" Sep 09 03:07:02.161503 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:02.161423 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qg2t\" (UniqueName: \"kubernetes.io/projected/cb150ea7-3521-4012-a358-d4abf3b2e2de-kube-api-access-6qg2t\") pod \"perf-node-gather-daemonset-p7hqs\" (UID: \"cb150ea7-3521-4012-a358-d4abf3b2e2de\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs" Sep 09 03:07:02.161503 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:02.161426 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb150ea7-3521-4012-a358-d4abf3b2e2de-proc\") pod \"perf-node-gather-daemonset-p7hqs\" (UID: \"cb150ea7-3521-4012-a358-d4abf3b2e2de\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs" Sep 09 03:07:02.161503 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:02.161425 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb150ea7-3521-4012-a358-d4abf3b2e2de-sys\") pod \"perf-node-gather-daemonset-p7hqs\" (UID: \"cb150ea7-3521-4012-a358-d4abf3b2e2de\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs" Sep 09 03:07:02.161503 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:02.161480 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb150ea7-3521-4012-a358-d4abf3b2e2de-lib-modules\") pod \"perf-node-gather-daemonset-p7hqs\" (UID: \"cb150ea7-3521-4012-a358-d4abf3b2e2de\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs" Sep 09 03:07:02.161503 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:02.161498 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb150ea7-3521-4012-a358-d4abf3b2e2de-podres\") pod \"perf-node-gather-daemonset-p7hqs\" (UID: \"cb150ea7-3521-4012-a358-d4abf3b2e2de\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs" Sep 09 03:07:02.171330 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:02.171309 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qg2t\" (UniqueName: \"kubernetes.io/projected/cb150ea7-3521-4012-a358-d4abf3b2e2de-kube-api-access-6qg2t\") pod \"perf-node-gather-daemonset-p7hqs\" (UID: \"cb150ea7-3521-4012-a358-d4abf3b2e2de\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs" Sep 09 03:07:02.249680 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:02.249637 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs" Sep 09 03:07:02.528211 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:02.527957 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs"] Sep 09 03:07:02.531018 ip-10-0-137-37 kubenswrapper[2577]: W0909 03:07:02.530988 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcb150ea7_3521_4012_a358_d4abf3b2e2de.slice/crio-efee20cfc114a15506737861999af51b57025f3e5a13226454a8358450ecf202 WatchSource:0}: Error finding container efee20cfc114a15506737861999af51b57025f3e5a13226454a8358450ecf202: Status 404 returned error can't find the container with id efee20cfc114a15506737861999af51b57025f3e5a13226454a8358450ecf202 Sep 09 03:07:02.960215 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:02.960176 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs" event={"ID":"cb150ea7-3521-4012-a358-d4abf3b2e2de","Type":"ContainerStarted","Data":"aaacd6e7304783d0be8227d172e8f53f5aa41d540304cf9fa3ea2cbc3bc72344"} Sep 09 03:07:02.960397 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:02.960229 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs" event={"ID":"cb150ea7-3521-4012-a358-d4abf3b2e2de","Type":"ContainerStarted","Data":"efee20cfc114a15506737861999af51b57025f3e5a13226454a8358450ecf202"} Sep 09 03:07:02.960397 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:02.960258 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs" Sep 09 03:07:02.982146 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:02.982094 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs" podStartSLOduration=1.982071178 podStartE2EDuration="1.982071178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:07:02.980148074 +0000 UTC m=+2211.110390169" watchObservedRunningTime="2026-09-09 03:07:02.982071178 +0000 UTC m=+2211.112313274" Sep 09 03:07:03.353751 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:03.353662 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-cz747_e03806d5-f065-4185-8899-a5b73b75befa/volume-data-source-validator/0.log" Sep 09 03:07:04.572385 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:04.572356 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2zpcz_35546a3a-6fe9-4cff-baae-84e4a07c527a/dns/0.log" Sep 09 03:07:04.598183 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:04.598158 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2zpcz_35546a3a-6fe9-4cff-baae-84e4a07c527a/kube-rbac-proxy/0.log" Sep 09 03:07:04.871008 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:04.870928 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wr95f_40631b35-00b4-4a4b-ab2e-f769434da1f1/dns-node-resolver/0.log" Sep 09 03:07:05.484448 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:05.484419 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qfvvm_2f959dac-774e-473c-b318-7cec8464e980/node-ca/0.log" Sep 09 03:07:06.853427 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:06.853396 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7dc95696d9-wz9vg_74f65ba8-8aae-4b5a-9229-0813168e1063/kube-auth-proxy/0.log" Sep 09 03:07:06.942432 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:06.942401 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-rggjg_7bd8d2bf-69f5-4f60-ae12-ff382fa47f7e/payload-pre-processing/0.log" Sep 09 03:07:07.888231 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:07.888197 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z9xdr_7aa94a6d-db29-4087-9416-92df99df14da/serve-healthcheck-canary/0.log" Sep 09 03:07:08.407777 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:08.407747 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mq79l_b5a72e3f-0e47-4305-b6f4-b9f294b90913/kube-rbac-proxy/0.log" Sep 09 03:07:08.425808 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:08.425785 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mq79l_b5a72e3f-0e47-4305-b6f4-b9f294b90913/exporter/0.log" Sep 09 03:07:08.444531 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:08.444501 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mq79l_b5a72e3f-0e47-4305-b6f4-b9f294b90913/extractor/0.log" Sep 09 03:07:08.976074 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:08.976048 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-p7hqs" Sep 09 03:07:10.923256 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:10.923227 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7849498579-zgmkn_5fd71c8a-79ad-4462-90a2-a75283e40cb2/manager/0.log" Sep 09 03:07:11.075246 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:11.075213 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dfc788bdd-4gjx2_0377c506-1c88-4580-bcfc-c81bd78e3f30/manager/0.log" Sep 09 03:07:11.095268 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:11.095207 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dfc788bdd-4gjx2_0377c506-1c88-4580-bcfc-c81bd78e3f30/copy-manifests/0.log" Sep 09 03:07:12.538103 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:12.538073 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6c4bb7846b-q5n4w_a2e31d9f-25f3-46c1-a37e-ddec8e1c92fd/manager/0.log" Sep 09 03:07:12.586788 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:12.586741 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-ct2gb_f4cec45e-ad48-4159-8ce3-5078d583e956/openshift-lws-operator/0.log" Sep 09 03:07:12.653390 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:12.653364 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7dc95696d9-wz9vg_74f65ba8-8aae-4b5a-9229-0813168e1063/kube-auth-proxy/0.log" Sep 09 03:07:12.725912 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:12.725880 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-rggjg_7bd8d2bf-69f5-4f60-ae12-ff382fa47f7e/payload-pre-processing/0.log" Sep 09 03:07:14.486954 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:14.486921 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7dc95696d9-wz9vg_74f65ba8-8aae-4b5a-9229-0813168e1063/kube-auth-proxy/0.log" Sep 09 03:07:14.557472 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:14.557441 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-rggjg_7bd8d2bf-69f5-4f60-ae12-ff382fa47f7e/payload-pre-processing/0.log" Sep 09 03:07:17.543627 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:17.543593 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-mf8qc_d948a89d-7114-420b-a168-c6fdc6ba64e1/migrator/0.log" Sep 09 03:07:17.561740 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:17.561712 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-mf8qc_d948a89d-7114-420b-a168-c6fdc6ba64e1/graceful-termination/0.log" Sep 09 03:07:18.939368 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:18.939340 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9pqdb_4b77648c-d60e-4ba3-806f-ea2f240ac764/kube-multus/0.log" Sep 09 03:07:18.960687 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:18.960665 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2ddl4_7bddd2af-4d25-4c54-8f2d-800934a15ab2/kube-multus-additional-cni-plugins/0.log" Sep 09 03:07:18.981874 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:18.981843 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2ddl4_7bddd2af-4d25-4c54-8f2d-800934a15ab2/egress-router-binary-copy/0.log" Sep 09 03:07:19.001031 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:19.001013 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2ddl4_7bddd2af-4d25-4c54-8f2d-800934a15ab2/cni-plugins/0.log" Sep 09 03:07:19.021707 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:19.021683 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2ddl4_7bddd2af-4d25-4c54-8f2d-800934a15ab2/bond-cni-plugin/0.log" Sep 09 03:07:19.047792 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:19.047767 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2ddl4_7bddd2af-4d25-4c54-8f2d-800934a15ab2/routeoverride-cni/0.log" Sep 09 03:07:19.070213 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:19.070193 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2ddl4_7bddd2af-4d25-4c54-8f2d-800934a15ab2/whereabouts-cni-bincopy/0.log" Sep 09 03:07:19.111322 ip-10-0-137-37 kubenswrapper[2577]: I0909 03:07:19.111300 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2ddl4_7bddd2af-4d25-4c54-8f2d-800934a15ab2/whereabouts-cni/0.log"