Sep 08 16:50:37.138706 ip-10-0-137-108 systemd[1]: Starting Kubernetes Kubelet... Sep 08 16:50:37.574181 ip-10-0-137-108 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 16:50:37.574181 ip-10-0-137-108 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 08 16:50:37.574181 ip-10-0-137-108 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 16:50:37.574181 ip-10-0-137-108 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 08 16:50:37.574181 ip-10-0-137-108 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 16:50:37.576647 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.576561 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 08 16:50:37.582296 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582281 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:50:37.582296 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582295 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:50:37.582359 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582300 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:50:37.582359 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582305 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:50:37.582359 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582310 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:50:37.582359 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582313 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:50:37.582359 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582317 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:50:37.582359 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582320 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:50:37.582359 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582323 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:50:37.582359 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582326 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:50:37.582359 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582329 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:50:37.582359 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582332 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:50:37.582359 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582334 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:50:37.582359 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582337 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:50:37.582359 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582340 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:50:37.582359 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582343 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:50:37.582359 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582345 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:50:37.582359 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582356 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:50:37.582359 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582359 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:50:37.582359 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582362 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:50:37.582359 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582365 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:50:37.582822 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582368 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:50:37.582822 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582370 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:50:37.582822 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582373 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:50:37.582822 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582376 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:50:37.582822 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582380 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:50:37.582822 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582382 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:50:37.582822 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582385 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:50:37.582822 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582387 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:50:37.582822 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582390 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:50:37.582822 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582392 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:50:37.582822 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582395 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:50:37.582822 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582397 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:50:37.582822 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582400 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:50:37.582822 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582402 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:50:37.582822 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582405 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:50:37.582822 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582407 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:50:37.582822 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582409 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:50:37.582822 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582412 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:50:37.582822 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582415 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:50:37.582822 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582417 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:50:37.583306 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582430 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:50:37.583306 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582433 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:50:37.583306 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582435 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:50:37.583306 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582438 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:50:37.583306 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582440 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:50:37.583306 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582443 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:50:37.583306 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582446 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:50:37.583306 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582449 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:50:37.583306 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582452 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:50:37.583306 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582454 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:50:37.583306 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582457 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:50:37.583306 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582460 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:50:37.583306 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582462 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:50:37.583306 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582466 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:50:37.583306 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582468 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:50:37.583306 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582471 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:50:37.583306 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582474 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:50:37.583306 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582476 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:50:37.583306 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582479 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:50:37.583851 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582481 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:50:37.583851 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582484 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:50:37.583851 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582486 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:50:37.583851 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582489 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:50:37.583851 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582492 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:50:37.583851 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582495 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:50:37.583851 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582498 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:50:37.583851 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582500 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:50:37.583851 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582503 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:50:37.583851 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582505 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:50:37.583851 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582508 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:50:37.583851 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582511 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:50:37.583851 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582514 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:50:37.583851 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582516 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:50:37.583851 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582519 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:50:37.583851 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582522 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:50:37.583851 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582526 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:50:37.583851 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582529 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:50:37.583851 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582532 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:50:37.583851 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582535 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:50:37.584324 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582541 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:50:37.584324 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582543 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:50:37.584324 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582546 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:50:37.584324 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582549 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:50:37.584324 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582551 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:50:37.584324 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582554 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:50:37.584324 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582557 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:50:37.584324 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582559 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:50:37.584324 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582562 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:50:37.584324 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582564 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:50:37.584324 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582567 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:50:37.584324 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582569 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:50:37.584324 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582572 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:50:37.584324 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582574 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:50:37.584324 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582577 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:50:37.584324 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582580 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:50:37.584324 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582582 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:50:37.584324 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582586 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:50:37.584324 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.582590 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:50:37.585948 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.585935 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:50:37.585948 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.585947 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:50:37.586008 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.585952 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:50:37.586008 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.585956 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:50:37.586008 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.585960 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:50:37.586008 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.585963 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:50:37.586008 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.585966 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:50:37.586008 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.585969 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:50:37.586008 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.585972 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:50:37.586008 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.585975 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:50:37.586008 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.585977 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:50:37.586008 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.585980 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:50:37.586008 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.585982 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:50:37.586008 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.585985 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:50:37.586008 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.585989 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:50:37.586008 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.585991 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:50:37.586008 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.585994 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:50:37.586008 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.585996 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:50:37.586008 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.585999 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:50:37.586008 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586001 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:50:37.586008 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586004 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:50:37.586514 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586008 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:50:37.586514 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586010 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:50:37.586514 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586013 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:50:37.586514 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586016 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:50:37.586514 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586019 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:50:37.586514 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586022 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:50:37.586514 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586024 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:50:37.586514 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586026 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:50:37.586514 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586029 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:50:37.586514 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586031 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:50:37.586514 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586034 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:50:37.586514 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586036 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:50:37.586514 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586039 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:50:37.586514 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586042 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:50:37.586514 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586044 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:50:37.586514 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586047 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:50:37.586514 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586049 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:50:37.586514 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586051 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:50:37.586514 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586054 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:50:37.586514 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586057 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:50:37.586999 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586059 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:50:37.586999 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586061 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:50:37.586999 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586064 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:50:37.586999 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586066 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:50:37.586999 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586069 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:50:37.586999 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586073 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:50:37.586999 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586076 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:50:37.586999 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586087 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:50:37.586999 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586090 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:50:37.586999 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586092 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:50:37.586999 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586095 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:50:37.586999 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586098 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:50:37.586999 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586102 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:50:37.586999 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586105 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:50:37.586999 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586109 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:50:37.586999 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586113 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:50:37.586999 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586115 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:50:37.586999 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586118 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:50:37.586999 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586120 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:50:37.587478 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586123 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:50:37.587478 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586125 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:50:37.587478 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586128 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:50:37.587478 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586130 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:50:37.587478 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586133 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:50:37.587478 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586136 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:50:37.587478 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586139 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:50:37.587478 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586141 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:50:37.587478 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586144 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:50:37.587478 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586146 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:50:37.587478 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586149 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:50:37.587478 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586151 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:50:37.587478 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586154 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:50:37.587478 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586156 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:50:37.587478 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586158 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:50:37.587478 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586161 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:50:37.587478 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586163 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:50:37.587478 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586166 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:50:37.587478 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586168 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:50:37.587478 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586170 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:50:37.588132 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586174 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:50:37.588132 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586176 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:50:37.588132 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586179 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:50:37.588132 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586181 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:50:37.588132 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586184 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:50:37.588132 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586187 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:50:37.588132 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586190 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:50:37.588132 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586192 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:50:37.588132 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586195 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:50:37.588132 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586197 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:50:37.588132 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586200 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:50:37.588132 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586202 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:50:37.588132 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586205 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:50:37.588132 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586207 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:50:37.588132 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586210 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:50:37.588132 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586212 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:50:37.588132 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586215 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:50:37.588132 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586217 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:50:37.588132 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586220 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:50:37.588132 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586582 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:50:37.588722 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586586 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:50:37.588722 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586589 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:50:37.588722 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586592 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:50:37.588722 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586595 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:50:37.588722 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586597 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:50:37.588722 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586600 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:50:37.588722 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586603 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:50:37.588722 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586606 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:50:37.588722 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586608 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:50:37.588722 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586611 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:50:37.588722 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586613 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:50:37.588722 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586616 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:50:37.588722 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586619 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:50:37.588722 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586622 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:50:37.588722 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586626 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:50:37.588722 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586630 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:50:37.588722 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586633 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:50:37.588722 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586637 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:50:37.588722 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586640 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:50:37.588722 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586643 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:50:37.589229 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586645 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:50:37.589229 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586648 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:50:37.589229 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586651 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:50:37.589229 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586654 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:50:37.589229 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586657 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:50:37.589229 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586659 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:50:37.589229 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586662 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:50:37.589229 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586664 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:50:37.589229 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586667 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:50:37.589229 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586670 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:50:37.589229 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586672 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:50:37.589229 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586675 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:50:37.589229 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586677 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:50:37.589229 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586680 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:50:37.589229 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586682 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:50:37.589229 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586685 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:50:37.589229 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586688 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:50:37.589229 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586690 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:50:37.589229 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586693 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:50:37.589229 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586695 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:50:37.589741 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586698 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:50:37.589741 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586700 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:50:37.589741 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586702 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:50:37.589741 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586706 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:50:37.589741 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586708 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:50:37.589741 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586711 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:50:37.589741 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586714 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:50:37.589741 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586717 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:50:37.589741 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586719 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:50:37.589741 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586722 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:50:37.589741 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586724 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:50:37.589741 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586727 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:50:37.589741 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586729 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:50:37.589741 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586732 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:50:37.589741 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586734 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:50:37.589741 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586737 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:50:37.589741 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586739 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:50:37.589741 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586742 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:50:37.589741 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586744 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:50:37.589741 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586746 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:50:37.590232 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586749 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:50:37.590232 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586751 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:50:37.590232 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586754 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:50:37.590232 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586756 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:50:37.590232 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586759 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:50:37.590232 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586761 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:50:37.590232 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586764 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:50:37.590232 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586766 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:50:37.590232 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586770 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:50:37.590232 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586773 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:50:37.590232 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586776 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:50:37.590232 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586779 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:50:37.590232 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586782 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:50:37.590232 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586785 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:50:37.590232 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586789 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:50:37.590232 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586793 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:50:37.590232 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586796 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:50:37.590232 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586799 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:50:37.590723 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586802 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:50:37.590723 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586804 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:50:37.590723 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586807 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:50:37.590723 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586810 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:50:37.590723 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586812 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:50:37.590723 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586815 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:50:37.590723 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586817 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:50:37.590723 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586820 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:50:37.590723 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586822 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:50:37.590723 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586825 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:50:37.590723 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586827 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:50:37.590723 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586830 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:50:37.590723 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586833 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:50:37.590723 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586835 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:50:37.590723 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586838 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:50:37.590723 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586840 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:50:37.590723 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586843 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:50:37.590723 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586845 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:50:37.590723 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586848 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:50:37.590723 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.586850 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.586930 2565 flags.go:64] FLAG: --address="0.0.0.0" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.586966 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.586972 2565 flags.go:64] FLAG: --anonymous-auth="true" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.586977 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.586982 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.586985 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.586989 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.586993 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.586996 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587000 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587004 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587007 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587011 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587015 2565 flags.go:64] FLAG: --cgroup-root="" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587018 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587021 2565 flags.go:64] FLAG: --client-ca-file="" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587025 2565 flags.go:64] FLAG: --cloud-provider="external" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587028 2565 flags.go:64] FLAG: --cluster-dns="[]" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587032 2565 flags.go:64] FLAG: --cluster-domain="" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587035 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587038 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587041 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587045 2565 flags.go:64] FLAG: --container-log-max-files="5" Sep 08 16:50:37.591207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587049 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587052 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587055 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587059 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587062 2565 flags.go:64] FLAG: --contention-profiling="false" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587065 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587068 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587071 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587073 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587077 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587080 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587083 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587086 2565 flags.go:64] FLAG: --enable-load-reader="false" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587091 2565 flags.go:64] FLAG: --enable-server="true" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587094 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587099 2565 flags.go:64] FLAG: --event-burst="100" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587102 2565 flags.go:64] FLAG: --event-qps="50" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587105 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587108 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587111 2565 flags.go:64] FLAG: --eviction-hard="" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587115 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587118 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587121 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587125 2565 flags.go:64] FLAG: --eviction-soft="" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587128 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 08 16:50:37.591827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587131 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587133 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587136 2565 flags.go:64] FLAG: --experimental-mounter-path="" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587139 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587142 2565 flags.go:64] FLAG: --fail-swap-on="true" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587145 2565 flags.go:64] FLAG: --feature-gates="" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587148 2565 flags.go:64] FLAG: --file-check-frequency="20s" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587151 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587154 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587158 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587160 2565 flags.go:64] FLAG: --healthz-port="10248" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587163 2565 flags.go:64] FLAG: --help="false" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587166 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-137-108.ec2.internal" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587169 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587172 2565 flags.go:64] FLAG: --http-check-frequency="20s" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587175 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587178 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587181 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587184 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587187 2565 flags.go:64] FLAG: --image-service-endpoint="" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587191 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587194 2565 flags.go:64] FLAG: --kube-api-burst="100" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587197 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587200 2565 flags.go:64] FLAG: --kube-api-qps="50" Sep 08 16:50:37.592516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587203 2565 flags.go:64] FLAG: --kube-reserved="" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587206 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587208 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587211 2565 flags.go:64] FLAG: --kubelet-cgroups="" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587214 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587217 2565 flags.go:64] FLAG: --lock-file="" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587220 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587223 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587226 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587232 2565 flags.go:64] FLAG: --log-json-split-stream="false" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587234 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587237 2565 flags.go:64] FLAG: --log-text-split-stream="false" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587240 2565 flags.go:64] FLAG: --logging-format="text" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587243 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587246 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587248 2565 flags.go:64] FLAG: --manifest-url="" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587251 2565 flags.go:64] FLAG: --manifest-url-header="" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587255 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587258 2565 flags.go:64] FLAG: --max-open-files="1000000" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587262 2565 flags.go:64] FLAG: --max-pods="110" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587265 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587268 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587270 2565 flags.go:64] FLAG: --memory-manager-policy="None" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587273 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587276 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 08 16:50:37.593119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587279 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587282 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587288 2565 flags.go:64] FLAG: --node-status-max-images="50" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587292 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587297 2565 flags.go:64] FLAG: --oom-score-adj="-999" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587300 2565 flags.go:64] FLAG: --pod-cidr="" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587303 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587309 2565 flags.go:64] FLAG: --pod-manifest-path="" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587311 2565 flags.go:64] FLAG: --pod-max-pids="-1" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587314 2565 flags.go:64] FLAG: --pods-per-core="0" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587317 2565 flags.go:64] FLAG: --port="10250" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587320 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587323 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-040e02575d1157f5b" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587326 2565 flags.go:64] FLAG: --qos-reserved="" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587329 2565 flags.go:64] FLAG: --read-only-port="10255" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587333 2565 flags.go:64] FLAG: --register-node="true" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587336 2565 flags.go:64] FLAG: --register-with-taints="" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587339 2565 flags.go:64] FLAG: --registry-burst="10" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587342 2565 flags.go:64] FLAG: --registry-qps="5" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587345 2565 flags.go:64] FLAG: --reserved-cpus="" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587348 2565 flags.go:64] FLAG: --reserved-memory="" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587351 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587354 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587357 2565 flags.go:64] FLAG: --rotate-certificates="false" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587360 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 08 16:50:37.593737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587363 2565 flags.go:64] FLAG: --runonce="false" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587365 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587368 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587371 2565 flags.go:64] FLAG: --seccomp-default="false" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587374 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587377 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587380 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587384 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587386 2565 flags.go:64] FLAG: --storage-driver-password="root" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587389 2565 flags.go:64] FLAG: --storage-driver-secure="false" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587392 2565 flags.go:64] FLAG: --storage-driver-table="stats" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587398 2565 flags.go:64] FLAG: --storage-driver-user="root" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587400 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587403 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587406 2565 flags.go:64] FLAG: --system-cgroups="" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587409 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587414 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587417 2565 flags.go:64] FLAG: --tls-cert-file="" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587433 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587437 2565 flags.go:64] FLAG: --tls-min-version="" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587440 2565 flags.go:64] FLAG: --tls-private-key-file="" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587443 2565 flags.go:64] FLAG: --topology-manager-policy="none" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587446 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587449 2565 flags.go:64] FLAG: --topology-manager-scope="container" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587452 2565 flags.go:64] FLAG: --v="2" Sep 08 16:50:37.594368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587456 2565 flags.go:64] FLAG: --version="false" Sep 08 16:50:37.595003 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587460 2565 flags.go:64] FLAG: --vmodule="" Sep 08 16:50:37.595003 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587463 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 08 16:50:37.595003 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.587467 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 08 16:50:37.595003 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588786 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:50:37.595003 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588800 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:50:37.595003 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588804 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:50:37.595003 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588807 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:50:37.595003 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588810 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:50:37.595003 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588813 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:50:37.595003 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588816 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:50:37.595003 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588819 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:50:37.595003 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588824 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:50:37.595003 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588828 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:50:37.595003 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588831 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:50:37.595003 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588834 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:50:37.595003 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588837 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:50:37.595003 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588840 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:50:37.595003 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588843 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:50:37.595003 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588847 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:50:37.595540 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588850 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:50:37.595540 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588853 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:50:37.595540 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588855 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:50:37.595540 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588858 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:50:37.595540 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588864 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:50:37.595540 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588867 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:50:37.595540 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588869 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:50:37.595540 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588872 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:50:37.595540 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588874 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:50:37.595540 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588877 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:50:37.595540 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588880 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:50:37.595540 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588882 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:50:37.595540 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588885 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:50:37.595540 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588887 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:50:37.595540 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588890 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:50:37.595540 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588892 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:50:37.595540 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588895 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:50:37.595540 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588898 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:50:37.595540 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588901 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:50:37.595540 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588904 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:50:37.596066 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588907 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:50:37.596066 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588909 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:50:37.596066 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588912 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:50:37.596066 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588914 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:50:37.596066 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588918 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:50:37.596066 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588921 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:50:37.596066 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588924 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:50:37.596066 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588927 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:50:37.596066 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588930 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:50:37.596066 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588933 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:50:37.596066 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588935 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:50:37.596066 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588938 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:50:37.596066 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588941 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:50:37.596066 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588947 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:50:37.596066 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588950 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:50:37.596066 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588953 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:50:37.596066 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588955 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:50:37.596066 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588958 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:50:37.596066 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588960 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:50:37.596066 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588963 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:50:37.596567 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588966 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:50:37.596567 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588969 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:50:37.596567 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588971 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:50:37.596567 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588974 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:50:37.596567 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588976 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:50:37.596567 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588979 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:50:37.596567 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588981 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:50:37.596567 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588984 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:50:37.596567 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588987 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:50:37.596567 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588990 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:50:37.596567 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588993 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:50:37.596567 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.588996 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:50:37.596567 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589000 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:50:37.596567 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589005 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:50:37.596567 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589008 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:50:37.596567 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589011 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:50:37.596567 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589013 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:50:37.596567 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589016 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:50:37.596567 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589019 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:50:37.597030 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589022 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:50:37.597030 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589025 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:50:37.597030 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589027 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:50:37.597030 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589030 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:50:37.597030 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589032 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:50:37.597030 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589035 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:50:37.597030 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589039 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:50:37.597030 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589042 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:50:37.597030 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589046 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:50:37.597030 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589049 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:50:37.597030 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589051 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:50:37.597030 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589054 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:50:37.597030 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589057 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:50:37.597030 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589059 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:50:37.597030 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589062 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:50:37.597030 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589064 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:50:37.597030 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589067 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:50:37.597030 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589069 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:50:37.597030 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589072 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:50:37.597030 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589075 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:50:37.597542 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589077 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:50:37.597542 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589080 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:50:37.597542 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589083 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:50:37.597542 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.589086 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:50:37.597542 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.590003 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 16:50:37.597675 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.597604 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 08 16:50:37.597675 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.597619 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 08 16:50:37.598597 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.598586 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 08 16:50:37.598636 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.598597 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 08 16:50:37.598669 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598653 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:50:37.598669 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598659 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:50:37.598669 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598663 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:50:37.598669 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598666 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:50:37.598669 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598670 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:50:37.598802 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598673 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:50:37.598802 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598676 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:50:37.598802 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598679 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:50:37.598802 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598682 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:50:37.598802 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598685 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:50:37.598802 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598688 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:50:37.598802 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598690 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:50:37.598802 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598693 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:50:37.598802 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598695 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:50:37.598802 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598698 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:50:37.598802 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598701 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:50:37.598802 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598703 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:50:37.598802 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598707 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:50:37.598802 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598709 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:50:37.598802 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598712 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:50:37.598802 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598714 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:50:37.598802 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598717 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:50:37.598802 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598719 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:50:37.598802 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598722 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:50:37.598802 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598724 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:50:37.599291 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598727 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:50:37.599291 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598729 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:50:37.599291 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598732 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:50:37.599291 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598735 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:50:37.599291 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598738 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:50:37.599291 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598741 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:50:37.599291 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598744 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:50:37.599291 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598746 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:50:37.599291 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598749 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:50:37.599291 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598751 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:50:37.599291 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598754 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:50:37.599291 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598756 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:50:37.599291 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598759 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:50:37.599291 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598762 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:50:37.599291 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598765 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:50:37.599291 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598767 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:50:37.599291 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598770 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:50:37.599291 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598773 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:50:37.599291 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598775 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:50:37.599291 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598778 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:50:37.599841 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598781 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:50:37.599841 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598783 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:50:37.599841 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598786 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:50:37.599841 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598788 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:50:37.599841 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598791 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:50:37.599841 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598793 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:50:37.599841 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598796 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:50:37.599841 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598798 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:50:37.599841 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598801 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:50:37.599841 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598803 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:50:37.599841 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598806 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:50:37.599841 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598809 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:50:37.599841 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598811 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:50:37.599841 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598814 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:50:37.599841 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598817 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:50:37.599841 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598819 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:50:37.599841 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598822 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:50:37.599841 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598826 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:50:37.599841 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598830 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:50:37.599841 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598833 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:50:37.600317 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598836 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:50:37.600317 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598839 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:50:37.600317 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598842 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:50:37.600317 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598844 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:50:37.600317 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598847 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:50:37.600317 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598850 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:50:37.600317 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598854 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:50:37.600317 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598858 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:50:37.600317 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598861 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:50:37.600317 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598864 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:50:37.600317 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598867 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:50:37.600317 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598870 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:50:37.600317 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598873 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:50:37.600317 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598876 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:50:37.600317 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598878 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:50:37.600317 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598880 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:50:37.600317 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598883 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:50:37.600317 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598885 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:50:37.600317 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598888 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:50:37.600796 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598890 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:50:37.600796 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598893 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:50:37.600796 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598895 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:50:37.600796 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598898 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:50:37.600796 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598900 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:50:37.600796 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598903 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:50:37.600796 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598905 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:50:37.600796 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598908 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:50:37.600796 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598910 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:50:37.600796 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598913 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:50:37.600796 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598916 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:50:37.600796 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598919 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:50:37.600796 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598922 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:50:37.600796 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598924 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:50:37.600796 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.598927 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:50:37.601156 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.598932 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 16:50:37.601156 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599038 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:50:37.601156 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599042 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:50:37.601156 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599046 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:50:37.601156 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599049 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:50:37.601156 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599052 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:50:37.601156 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599055 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:50:37.601156 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599058 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:50:37.601156 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599060 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:50:37.601156 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599063 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:50:37.601156 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599066 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:50:37.601156 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599068 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:50:37.601156 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599071 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:50:37.601156 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599074 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:50:37.601156 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599077 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:50:37.601543 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599080 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:50:37.601543 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599082 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:50:37.601543 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599085 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:50:37.601543 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599087 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:50:37.601543 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599090 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:50:37.601543 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599093 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:50:37.601543 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599095 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:50:37.601543 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599099 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:50:37.601543 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599103 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:50:37.601543 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599106 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:50:37.601543 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599109 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:50:37.601543 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599111 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:50:37.601543 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599114 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:50:37.601543 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599116 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:50:37.601543 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599120 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:50:37.601543 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599123 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:50:37.601543 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599125 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:50:37.601543 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599128 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:50:37.601543 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599130 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:50:37.601543 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599133 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:50:37.602032 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599136 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:50:37.602032 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599138 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:50:37.602032 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599141 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:50:37.602032 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599143 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:50:37.602032 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599146 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:50:37.602032 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599148 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:50:37.602032 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599151 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:50:37.602032 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599153 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:50:37.602032 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599155 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:50:37.602032 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599158 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:50:37.602032 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599160 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:50:37.602032 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599163 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:50:37.602032 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599166 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:50:37.602032 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599168 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:50:37.602032 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599171 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:50:37.602032 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599173 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:50:37.602032 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599176 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:50:37.602032 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599178 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:50:37.602032 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599181 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:50:37.602032 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599184 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:50:37.602603 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599186 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:50:37.602603 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599189 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:50:37.602603 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599191 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:50:37.602603 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599194 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:50:37.602603 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599196 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:50:37.602603 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599199 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:50:37.602603 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599201 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:50:37.602603 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599204 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:50:37.602603 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599207 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:50:37.602603 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599210 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:50:37.602603 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599212 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:50:37.602603 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599215 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:50:37.602603 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599218 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:50:37.602603 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599220 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:50:37.602603 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599223 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:50:37.602603 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599225 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:50:37.602603 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599228 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:50:37.602603 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599230 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:50:37.602603 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599233 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:50:37.603054 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599235 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:50:37.603054 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599238 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:50:37.603054 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599240 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:50:37.603054 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599243 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:50:37.603054 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599245 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:50:37.603054 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599248 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:50:37.603054 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599250 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:50:37.603054 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599253 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:50:37.603054 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599255 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:50:37.603054 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599258 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:50:37.603054 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599260 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:50:37.603054 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599264 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:50:37.603054 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599268 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:50:37.603054 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599271 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:50:37.603054 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599274 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:50:37.603054 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599277 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:50:37.603054 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599279 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:50:37.603054 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599282 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:50:37.603054 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599284 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:50:37.603530 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599288 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:50:37.603530 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599291 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:50:37.603530 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599295 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:50:37.603530 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599299 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:50:37.603530 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599301 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:50:37.603530 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599305 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:50:37.603530 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:37.599308 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:50:37.603530 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.599312 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 16:50:37.603530 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.599414 2565 server.go:962] "Client rotation is on, will bootstrap in background" Sep 08 16:50:37.603530 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.601907 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 08 16:50:37.603530 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.602884 2565 server.go:1019] "Starting client certificate rotation" Sep 08 16:50:37.603530 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.602987 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 16:50:37.603530 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.603023 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 16:50:37.605633 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.605619 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 16:50:37.612260 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.612238 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 16:50:37.622102 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.622082 2565 log.go:25] "Validated CRI v1 runtime API" Sep 08 16:50:37.626359 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.626342 2565 log.go:25] "Validated CRI v1 image API" Sep 08 16:50:37.627205 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.627190 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 08 16:50:37.633571 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.633552 2565 fs.go:135] Filesystem UUIDs: map[61182155-87ce-420b-bbce-394354b0a245:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ea3101cb-dc1a-4f1d-9bd1-2f4a3b27aa4d:/dev/nvme0n1p4] Sep 08 16:50:37.633649 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.633570 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 08 16:50:37.639556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.639446 2565 manager.go:216] Machine: {Timestamp:2026-09-08 16:50:37.63710474 +0000 UTC m=+0.373935846 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099876 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec213a671cd8d9ef1c512b3153611bb1 SystemUUID:ec213a67-1cd8-d9ef-1c51-2b3153611bb1 BootID:1b52e5d8-0963-4562-b4ab-8633fe49d735 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2d:20:80:08:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2d:20:80:08:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:22:b5:04:a0:51 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 08 16:50:37.639556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.639555 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 08 16:50:37.639665 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.639631 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 08 16:50:37.641393 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.641379 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 16:50:37.643438 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.643398 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 08 16:50:37.643586 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.643441 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-108.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 08 16:50:37.644310 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.644301 2565 topology_manager.go:138] "Creating topology manager with none policy" Sep 08 16:50:37.644348 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.644312 2565 container_manager_linux.go:309] "Creating device plugin manager" Sep 08 16:50:37.644348 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.644321 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 16:50:37.644348 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.644334 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 16:50:37.646315 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.646305 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 08 16:50:37.651026 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.651015 2565 state_mem.go:36] "Initialized new in-memory state store" Sep 08 16:50:37.651138 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.651128 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 08 16:50:37.651175 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.651157 2565 kubelet.go:486] "Attempting to sync node with API server" Sep 08 16:50:37.651175 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.651169 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 08 16:50:37.651228 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.651190 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 08 16:50:37.651228 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.651205 2565 kubelet.go:398] "Adding apiserver pod source" Sep 08 16:50:37.651228 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.651215 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 08 16:50:37.651573 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.651561 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 16:50:37.651612 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.651584 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 16:50:37.654666 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.654649 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 08 16:50:37.657639 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.657626 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 08 16:50:37.657691 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.657655 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 08 16:50:37.657819 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.657808 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 08 16:50:37.657849 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.657824 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 08 16:50:37.657849 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.657830 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 08 16:50:37.657849 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.657836 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 08 16:50:37.657849 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.657844 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 08 16:50:37.657849 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.657849 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 08 16:50:37.658464 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.658454 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 08 16:50:37.658464 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.658464 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 08 16:50:37.658519 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.658470 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 08 16:50:37.658519 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.658476 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 08 16:50:37.658519 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.658484 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 08 16:50:37.658519 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.658492 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 08 16:50:37.660363 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.660344 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 08 16:50:37.660363 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.660362 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 08 16:50:37.662030 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:37.662009 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-108.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 08 16:50:37.662099 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:37.662010 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 08 16:50:37.664198 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.664185 2565 server.go:1268] "Started kubelet" Sep 08 16:50:37.664300 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.664261 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 08 16:50:37.664982 ip-10-0-137-108 systemd[1]: Started Kubernetes Kubelet. Sep 08 16:50:37.666168 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.666110 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 08 16:50:37.666250 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.666196 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 08 16:50:37.666512 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.666497 2565 server.go:310] "Adding debug handlers to kubelet server" Sep 08 16:50:37.667778 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.667759 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 08 16:50:37.669464 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:37.669383 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 08 16:50:37.672540 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.672518 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-108.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 08 16:50:37.674269 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.674248 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 08 16:50:37.674875 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:37.672477 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-108.ec2.internal.18d3668e238e939a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-108.ec2.internal,UID:ip-10-0-137-108.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-108.ec2.internal,},FirstTimestamp:2026-09-08 16:50:37.664154522 +0000 UTC m=+0.400985629,LastTimestamp:2026-09-08 16:50:37.664154522 +0000 UTC m=+0.400985629,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-108.ec2.internal,}" Sep 08 16:50:37.675019 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.674972 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 08 16:50:37.676092 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:37.675858 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-108.ec2.internal\" not found" Sep 08 16:50:37.676509 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.676492 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 08 16:50:37.676595 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.676513 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 08 16:50:37.676658 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.676623 2565 reconstruct.go:97] "Volume reconstruction finished" Sep 08 16:50:37.676658 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.676631 2565 reconciler.go:29] "Reconciler: start to sync state" Sep 08 16:50:37.676658 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.676640 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 08 16:50:37.676929 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.676906 2565 factory.go:55] Registering systemd factory Sep 08 16:50:37.676929 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.676923 2565 factory.go:223] Registration of the systemd container factory successfully Sep 08 16:50:37.677412 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.677403 2565 factory.go:153] Registering CRI-O factory Sep 08 16:50:37.677485 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.677418 2565 factory.go:223] Registration of the crio container factory successfully Sep 08 16:50:37.677558 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:37.677534 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 08 16:50:37.677614 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.677564 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 08 16:50:37.677614 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:37.677575 2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-108.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 08 16:50:37.677614 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.677588 2565 factory.go:103] Registering Raw factory Sep 08 16:50:37.677739 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.677615 2565 manager.go:1196] Started watching for new ooms in manager Sep 08 16:50:37.678013 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.677991 2565 manager.go:349] Starting recovery of all containers Sep 08 16:50:37.685930 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.685903 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kztnc" Sep 08 16:50:37.686755 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.686730 2565 manager.go:354] Recovery completed Sep 08 16:50:37.691562 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.691546 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:50:37.693235 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.693221 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-108.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:50:37.693311 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.693247 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-108.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:50:37.693311 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.693257 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-108.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:50:37.693605 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.693578 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kztnc" Sep 08 16:50:37.693862 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.693847 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 08 16:50:37.693862 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.693859 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 08 16:50:37.693967 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.693874 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:50:37.693967 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.693876 2565 state_mem.go:36] "Initialized new in-memory state store" Sep 08 16:50:37.694569 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.694550 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-108.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:50:37.694668 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.694576 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-108.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:50:37.694668 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.694585 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-108.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:50:37.696714 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.696701 2565 policy_none.go:49] "None policy: Start" Sep 08 16:50:37.696776 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.696722 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 08 16:50:37.696776 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.696733 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 08 16:50:37.698686 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.698675 2565 policy_none.go:47] "Start" Sep 08 16:50:37.751752 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.751731 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 08 16:50:37.762076 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.752932 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 08 16:50:37.762076 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.752985 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 08 16:50:37.762076 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.753009 2565 kubelet.go:2448] "Starting kubelet main sync loop" Sep 08 16:50:37.762076 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:37.753048 2565 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Sep 08 16:50:37.762076 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.755413 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:50:37.776364 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:37.776348 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-108.ec2.internal\" not found" Sep 08 16:50:37.853123 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:37.853101 2565 kubelet.go:2472] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Sep 08 16:50:37.870938 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.870921 2565 manager.go:337] "Starting Device Plugin manager" Sep 08 16:50:37.871010 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:37.870959 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 08 16:50:37.871010 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.870972 2565 server.go:87] "Starting device plugin registration server" Sep 08 16:50:37.871335 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.871324 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 08 16:50:37.871371 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.871337 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 08 16:50:37.871474 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.871460 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 08 16:50:37.871542 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.871532 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 08 16:50:37.871596 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.871543 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 08 16:50:37.872044 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:37.872027 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 08 16:50:37.872129 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:37.872070 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-108.ec2.internal\" not found" Sep 08 16:50:37.883571 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:37.883548 2565 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-108.ec2.internal\" not found" node="ip-10-0-137-108.ec2.internal" Sep 08 16:50:37.971708 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.971682 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:50:37.972462 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.972445 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-108.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:50:37.972526 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.972477 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-108.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:50:37.972526 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.972493 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-108.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:50:37.972526 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.972513 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-108.ec2.internal" Sep 08 16:50:37.982056 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:37.982040 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-108.ec2.internal" Sep 08 16:50:37.982139 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:37.982061 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-108.ec2.internal\": node \"ip-10-0-137-108.ec2.internal\" not found" Sep 08 16:50:38.012239 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:38.012219 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-108.ec2.internal\" not found" Sep 08 16:50:38.053361 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.053303 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-108.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-108.ec2.internal"] Sep 08 16:50:38.053442 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.053401 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:50:38.055052 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.055035 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-108.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:50:38.055123 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.055063 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-108.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:50:38.055123 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.055073 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-108.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:50:38.057310 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.057298 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:50:38.057416 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.057404 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-108.ec2.internal" Sep 08 16:50:38.057484 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.057444 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:50:38.057944 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.057929 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-108.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:50:38.057999 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.057953 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-108.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:50:38.057999 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.057975 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-108.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:50:38.057999 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.057987 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-108.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:50:38.058106 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.057956 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-108.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:50:38.058106 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.058043 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-108.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:50:38.060315 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.060298 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-108.ec2.internal" Sep 08 16:50:38.060412 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.060321 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:50:38.060940 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.060921 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-108.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:50:38.061023 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.060955 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-108.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:50:38.061023 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.060968 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-108.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:50:38.076933 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:38.076910 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-108.ec2.internal\" not found" node="ip-10-0-137-108.ec2.internal" Sep 08 16:50:38.078378 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.078360 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01139cbc3b4b1c8ef9cbefa8f1a92b6f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-108.ec2.internal\" (UID: \"01139cbc3b4b1c8ef9cbefa8f1a92b6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-108.ec2.internal" Sep 08 16:50:38.078469 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.078383 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/209ac989f5bdb50086e7e268153789b2-config\") pod \"kube-apiserver-proxy-ip-10-0-137-108.ec2.internal\" (UID: \"209ac989f5bdb50086e7e268153789b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-108.ec2.internal" Sep 08 16:50:38.078469 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.078407 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/01139cbc3b4b1c8ef9cbefa8f1a92b6f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-108.ec2.internal\" (UID: \"01139cbc3b4b1c8ef9cbefa8f1a92b6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-108.ec2.internal" Sep 08 16:50:38.080762 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:38.080745 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-108.ec2.internal\" not found" node="ip-10-0-137-108.ec2.internal" Sep 08 16:50:38.112971 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:38.112927 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-108.ec2.internal\" not found" Sep 08 16:50:38.178536 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.178502 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/01139cbc3b4b1c8ef9cbefa8f1a92b6f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-108.ec2.internal\" (UID: \"01139cbc3b4b1c8ef9cbefa8f1a92b6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-108.ec2.internal" Sep 08 16:50:38.178636 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.178546 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01139cbc3b4b1c8ef9cbefa8f1a92b6f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-108.ec2.internal\" (UID: \"01139cbc3b4b1c8ef9cbefa8f1a92b6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-108.ec2.internal" Sep 08 16:50:38.178636 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.178570 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/209ac989f5bdb50086e7e268153789b2-config\") pod \"kube-apiserver-proxy-ip-10-0-137-108.ec2.internal\" (UID: \"209ac989f5bdb50086e7e268153789b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-108.ec2.internal" Sep 08 16:50:38.178636 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.178609 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/01139cbc3b4b1c8ef9cbefa8f1a92b6f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-108.ec2.internal\" (UID: \"01139cbc3b4b1c8ef9cbefa8f1a92b6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-108.ec2.internal" Sep 08 16:50:38.178636 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.178622 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01139cbc3b4b1c8ef9cbefa8f1a92b6f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-108.ec2.internal\" (UID: \"01139cbc3b4b1c8ef9cbefa8f1a92b6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-108.ec2.internal" Sep 08 16:50:38.178636 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.178628 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/209ac989f5bdb50086e7e268153789b2-config\") pod \"kube-apiserver-proxy-ip-10-0-137-108.ec2.internal\" (UID: \"209ac989f5bdb50086e7e268153789b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-108.ec2.internal" Sep 08 16:50:38.213619 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:38.213604 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-108.ec2.internal\" not found" Sep 08 16:50:38.314064 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:38.314035 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-108.ec2.internal\" not found" Sep 08 16:50:38.381198 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.381136 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-108.ec2.internal" Sep 08 16:50:38.384492 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.384475 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-108.ec2.internal" Sep 08 16:50:38.414711 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:38.414678 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-108.ec2.internal\" not found" Sep 08 16:50:38.497879 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.497856 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:50:38.500408 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.500393 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:50:38.579801 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.579780 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-108.ec2.internal" Sep 08 16:50:38.588585 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.588570 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 16:50:38.590798 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.590785 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-108.ec2.internal" Sep 08 16:50:38.602549 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.602533 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 08 16:50:38.602663 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.602639 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 16:50:38.602711 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.602673 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 16:50:38.602711 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.602683 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 16:50:38.602711 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:38.602689 2565 request.go:1196] "Unexpected error when reading response body" err="read tcp 10.0.137.108:46264->98.82.74.88:6443: use of closed network connection" Sep 08 16:50:38.602813 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:38.602736 2565 kubelet.go:3250] "Failed creating a mirror pod" err="unexpected error when reading response body. Please retry. Original error: read tcp 10.0.137.108:46264->98.82.74.88:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-108.ec2.internal" Sep 08 16:50:38.653082 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.653018 2565 apiserver.go:52] "Watching apiserver" Sep 08 16:50:38.669272 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.669245 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 08 16:50:38.669635 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.669614 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-5qbl5","openshift-multus/network-metrics-daemon-qbb9j","openshift-network-operator/iptables-alerter-bbtgc","openshift-ovn-kubernetes/ovnkube-node-9xxnl","kube-system/global-pull-secret-syncer-jpx5h","openshift-cluster-node-tuning-operator/tuned-zdmwv","openshift-dns/node-resolver-2fv8p","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-108.ec2.internal","openshift-multus/multus-9lwh2","openshift-network-diagnostics/network-check-target-lzqb8","kube-system/konnectivity-agent-2ccgf","kube-system/kube-apiserver-proxy-ip-10-0-137-108.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58","openshift-image-registry/node-ca-rr4fp"] Sep 08 16:50:38.672313 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.672295 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2fv8p" Sep 08 16:50:38.674463 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.674407 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 08 16:50:38.674682 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.674661 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:38.674784 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.674693 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 08 16:50:38.674784 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.674769 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 08 16:50:38.674896 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.674830 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4prjq\"" Sep 08 16:50:38.676736 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.676716 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 08 16:50:38.676818 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.676775 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7kjj7\"" Sep 08 16:50:38.677039 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.677019 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bbtgc" Sep 08 16:50:38.677129 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.677061 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 08 16:50:38.677129 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.677124 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 08 16:50:38.679021 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.679000 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wv25c\"" Sep 08 16:50:38.679107 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.679063 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:50:38.679107 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.679063 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 08 16:50:38.679226 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.679212 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 08 16:50:38.679568 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.679551 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.681114 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.681086 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e88b3e9c-6397-4627-a317-933f2576b451-registration-dir\") pod \"aws-ebs-csi-driver-node-jms58\" (UID: \"e88b3e9c-6397-4627-a317-933f2576b451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:38.681221 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.681185 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e88b3e9c-6397-4627-a317-933f2576b451-etc-selinux\") pod \"aws-ebs-csi-driver-node-jms58\" (UID: \"e88b3e9c-6397-4627-a317-933f2576b451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:38.681311 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.681296 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lqll\" (UniqueName: \"kubernetes.io/projected/e88b3e9c-6397-4627-a317-933f2576b451-kube-api-access-5lqll\") pod \"aws-ebs-csi-driver-node-jms58\" (UID: \"e88b3e9c-6397-4627-a317-933f2576b451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:38.681374 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.681336 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxq29\" (UniqueName: \"kubernetes.io/projected/9abcfc03-c9b3-4457-95d8-cc2686ff907a-kube-api-access-nxq29\") pod \"iptables-alerter-bbtgc\" (UID: \"9abcfc03-c9b3-4457-95d8-cc2686ff907a\") " pod="openshift-network-operator/iptables-alerter-bbtgc" Sep 08 16:50:38.681431 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.681395 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1da361fa-6e95-4a71-bac4-e4e5a2c6d5ef-hosts-file\") pod \"node-resolver-2fv8p\" (UID: \"1da361fa-6e95-4a71-bac4-e4e5a2c6d5ef\") " pod="openshift-dns/node-resolver-2fv8p" Sep 08 16:50:38.681492 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.681449 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1da361fa-6e95-4a71-bac4-e4e5a2c6d5ef-tmp-dir\") pod \"node-resolver-2fv8p\" (UID: \"1da361fa-6e95-4a71-bac4-e4e5a2c6d5ef\") " pod="openshift-dns/node-resolver-2fv8p" Sep 08 16:50:38.681630 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.681507 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9xsb\" (UniqueName: \"kubernetes.io/projected/1da361fa-6e95-4a71-bac4-e4e5a2c6d5ef-kube-api-access-p9xsb\") pod \"node-resolver-2fv8p\" (UID: \"1da361fa-6e95-4a71-bac4-e4e5a2c6d5ef\") " pod="openshift-dns/node-resolver-2fv8p" Sep 08 16:50:38.681630 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.681547 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9abcfc03-c9b3-4457-95d8-cc2686ff907a-iptables-alerter-script\") pod \"iptables-alerter-bbtgc\" (UID: \"9abcfc03-c9b3-4457-95d8-cc2686ff907a\") " pod="openshift-network-operator/iptables-alerter-bbtgc" Sep 08 16:50:38.681630 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.681622 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e88b3e9c-6397-4627-a317-933f2576b451-sys-fs\") pod \"aws-ebs-csi-driver-node-jms58\" (UID: \"e88b3e9c-6397-4627-a317-933f2576b451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:38.681807 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.681661 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e88b3e9c-6397-4627-a317-933f2576b451-device-dir\") pod \"aws-ebs-csi-driver-node-jms58\" (UID: \"e88b3e9c-6397-4627-a317-933f2576b451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:38.681807 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.681711 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9abcfc03-c9b3-4457-95d8-cc2686ff907a-host-slash\") pod \"iptables-alerter-bbtgc\" (UID: \"9abcfc03-c9b3-4457-95d8-cc2686ff907a\") " pod="openshift-network-operator/iptables-alerter-bbtgc" Sep 08 16:50:38.681807 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.681752 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e88b3e9c-6397-4627-a317-933f2576b451-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jms58\" (UID: \"e88b3e9c-6397-4627-a317-933f2576b451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:38.681807 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.681792 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e88b3e9c-6397-4627-a317-933f2576b451-socket-dir\") pod \"aws-ebs-csi-driver-node-jms58\" (UID: \"e88b3e9c-6397-4627-a317-933f2576b451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:38.681995 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.681950 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 08 16:50:38.681995 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.681958 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 08 16:50:38.682254 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.682237 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 08 16:50:38.683353 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.682839 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:38.683353 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.682910 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 08 16:50:38.683353 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:38.682921 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jpx5h" podUID="192b1a00-b435-4793-98d5-d51cf7b036d1" Sep 08 16:50:38.683353 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.683067 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4rwhf\"" Sep 08 16:50:38.683353 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.683268 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 08 16:50:38.683353 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.683311 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 08 16:50:38.684562 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.684539 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 16:50:38.685630 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.685611 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.687669 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.687634 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 08 16:50:38.687766 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.687693 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qnjkd\"" Sep 08 16:50:38.687766 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.687747 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:50:38.688056 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.688038 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.690210 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.690190 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 08 16:50:38.690296 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.690254 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 08 16:50:38.690366 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.690297 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8pjnt\"" Sep 08 16:50:38.690366 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.690352 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 08 16:50:38.690526 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.690482 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 08 16:50:38.690738 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.690723 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:38.692620 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.692603 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 08 16:50:38.692989 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.692975 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:50:38.693073 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:38.693050 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb9j" podUID="5d001365-79a1-4fec-bcf8-3abf7309f1ff" Sep 08 16:50:38.693274 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.693250 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 08 16:50:38.693356 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.693301 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5k7wg\"" Sep 08 16:50:38.695411 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.695397 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:50:38.695481 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:38.695455 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzqb8" podUID="e59c8379-7774-4209-bda5-cd8c13a1e464" Sep 08 16:50:38.697231 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.697195 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 16:45:38 +0000 UTC" deadline="2028-03-25 03:39:39.914776933 +0000 UTC" Sep 08 16:50:38.697231 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.697230 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13522h49m1.217549241s" Sep 08 16:50:38.698032 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.698014 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2ccgf" Sep 08 16:50:38.700480 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.700455 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 08 16:50:38.700480 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.700468 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-z5ngt\"" Sep 08 16:50:38.700480 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.700477 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 08 16:50:38.700672 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.700653 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rr4fp" Sep 08 16:50:38.702841 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.702809 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 08 16:50:38.702933 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.702867 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 08 16:50:38.703080 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.703063 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 08 16:50:38.703170 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.703092 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4w25c\"" Sep 08 16:50:38.715091 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.715073 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-j6knp" Sep 08 16:50:38.724916 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.724890 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j6knp" Sep 08 16:50:38.780255 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.778735 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 08 16:50:38.781973 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.781941 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e88b3e9c-6397-4627-a317-933f2576b451-socket-dir\") pod \"aws-ebs-csi-driver-node-jms58\" (UID: \"e88b3e9c-6397-4627-a317-933f2576b451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:38.782074 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.781976 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e88b3e9c-6397-4627-a317-933f2576b451-registration-dir\") pod \"aws-ebs-csi-driver-node-jms58\" (UID: \"e88b3e9c-6397-4627-a317-933f2576b451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:38.782074 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.781996 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lqll\" (UniqueName: \"kubernetes.io/projected/e88b3e9c-6397-4627-a317-933f2576b451-kube-api-access-5lqll\") pod \"aws-ebs-csi-driver-node-jms58\" (UID: \"e88b3e9c-6397-4627-a317-933f2576b451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:38.782074 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782016 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxq29\" (UniqueName: \"kubernetes.io/projected/9abcfc03-c9b3-4457-95d8-cc2686ff907a-kube-api-access-nxq29\") pod \"iptables-alerter-bbtgc\" (UID: \"9abcfc03-c9b3-4457-95d8-cc2686ff907a\") " pod="openshift-network-operator/iptables-alerter-bbtgc" Sep 08 16:50:38.782074 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782043 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-log-socket\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.782074 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782069 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-etc-sysctl-d\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.782337 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782083 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e88b3e9c-6397-4627-a317-933f2576b451-registration-dir\") pod \"aws-ebs-csi-driver-node-jms58\" (UID: \"e88b3e9c-6397-4627-a317-933f2576b451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:38.782337 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782093 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-hostroot\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.782337 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782127 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1da361fa-6e95-4a71-bac4-e4e5a2c6d5ef-hosts-file\") pod \"node-resolver-2fv8p\" (UID: \"1da361fa-6e95-4a71-bac4-e4e5a2c6d5ef\") " pod="openshift-dns/node-resolver-2fv8p" Sep 08 16:50:38.782337 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782150 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-host-cni-netd\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.782337 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782166 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/92dbe1ae-0e8e-4772-9c97-be594f062d76-serviceca\") pod \"node-ca-rr4fp\" (UID: \"92dbe1ae-0e8e-4772-9c97-be594f062d76\") " pod="openshift-image-registry/node-ca-rr4fp" Sep 08 16:50:38.782337 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782181 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-host-run-netns\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.782337 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782186 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1da361fa-6e95-4a71-bac4-e4e5a2c6d5ef-hosts-file\") pod \"node-resolver-2fv8p\" (UID: \"1da361fa-6e95-4a71-bac4-e4e5a2c6d5ef\") " pod="openshift-dns/node-resolver-2fv8p" Sep 08 16:50:38.782337 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782202 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-host\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.782337 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782228 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e88b3e9c-6397-4627-a317-933f2576b451-device-dir\") pod \"aws-ebs-csi-driver-node-jms58\" (UID: \"e88b3e9c-6397-4627-a317-933f2576b451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:38.782337 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782243 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-host-cni-bin\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.782337 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782257 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf53ac63-f864-402b-bf12-45b289e7e13d-ovnkube-config\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.782337 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782272 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-var-lib-kubelet\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.782337 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782292 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-system-cni-dir\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.782337 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782307 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e88b3e9c-6397-4627-a317-933f2576b451-device-dir\") pod \"aws-ebs-csi-driver-node-jms58\" (UID: \"e88b3e9c-6397-4627-a317-933f2576b451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:38.782337 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782313 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e88b3e9c-6397-4627-a317-933f2576b451-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jms58\" (UID: \"e88b3e9c-6397-4627-a317-933f2576b451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:38.782337 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782342 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-etc-modprobe-d\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.782915 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782363 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwtbq\" (UniqueName: \"kubernetes.io/projected/15b7eb50-d4dc-4794-9448-164cf91390a9-kube-api-access-vwtbq\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.782915 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782387 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-etc-kubernetes\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.782915 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782396 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e88b3e9c-6397-4627-a317-933f2576b451-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jms58\" (UID: \"e88b3e9c-6397-4627-a317-933f2576b451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:38.782915 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782412 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs\") pod \"network-metrics-daemon-qbb9j\" (UID: \"5d001365-79a1-4fec-bcf8-3abf7309f1ff\") " pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:50:38.782915 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782436 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1da361fa-6e95-4a71-bac4-e4e5a2c6d5ef-tmp-dir\") pod \"node-resolver-2fv8p\" (UID: \"1da361fa-6e95-4a71-bac4-e4e5a2c6d5ef\") " pod="openshift-dns/node-resolver-2fv8p" Sep 08 16:50:38.782915 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782469 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-etc-openvswitch\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.782915 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782504 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6275fc1b-99a7-413c-a0fe-628da6828cd1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5qbl5\" (UID: \"6275fc1b-99a7-413c-a0fe-628da6828cd1\") " pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:38.782915 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782532 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-etc-sysconfig\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.782915 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782557 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-host-run-k8s-cni-cncf-io\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.782915 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782579 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-host-run-netns\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.782915 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782614 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-multus-conf-dir\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.782915 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782647 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e88b3e9c-6397-4627-a317-933f2576b451-sys-fs\") pod \"aws-ebs-csi-driver-node-jms58\" (UID: \"e88b3e9c-6397-4627-a317-933f2576b451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:38.782915 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782672 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-node-log\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.782915 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782695 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-sys\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.782915 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782717 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e88b3e9c-6397-4627-a317-933f2576b451-sys-fs\") pod \"aws-ebs-csi-driver-node-jms58\" (UID: \"e88b3e9c-6397-4627-a317-933f2576b451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:38.782915 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782719 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-cni-binary-copy\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.782915 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782760 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-systemd-units\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.783546 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782783 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf53ac63-f864-402b-bf12-45b289e7e13d-ovnkube-script-lib\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.783546 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782798 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/15b7eb50-d4dc-4794-9448-164cf91390a9-etc-tuned\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.783546 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782761 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1da361fa-6e95-4a71-bac4-e4e5a2c6d5ef-tmp-dir\") pod \"node-resolver-2fv8p\" (UID: \"1da361fa-6e95-4a71-bac4-e4e5a2c6d5ef\") " pod="openshift-dns/node-resolver-2fv8p" Sep 08 16:50:38.783546 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782814 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n56p\" (UniqueName: \"kubernetes.io/projected/5d001365-79a1-4fec-bcf8-3abf7309f1ff-kube-api-access-9n56p\") pod \"network-metrics-daemon-qbb9j\" (UID: \"5d001365-79a1-4fec-bcf8-3abf7309f1ff\") " pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:50:38.783546 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782826 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e88b3e9c-6397-4627-a317-933f2576b451-socket-dir\") pod \"aws-ebs-csi-driver-node-jms58\" (UID: \"e88b3e9c-6397-4627-a317-933f2576b451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:38.783546 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782834 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e88b3e9c-6397-4627-a317-933f2576b451-etc-selinux\") pod \"aws-ebs-csi-driver-node-jms58\" (UID: \"e88b3e9c-6397-4627-a317-933f2576b451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:38.783546 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782875 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e88b3e9c-6397-4627-a317-933f2576b451-etc-selinux\") pod \"aws-ebs-csi-driver-node-jms58\" (UID: \"e88b3e9c-6397-4627-a317-933f2576b451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:38.783546 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782905 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/192b1a00-b435-4793-98d5-d51cf7b036d1-kubelet-config\") pod \"global-pull-secret-syncer-jpx5h\" (UID: \"192b1a00-b435-4793-98d5-d51cf7b036d1\") " pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:38.783546 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782933 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6275fc1b-99a7-413c-a0fe-628da6828cd1-system-cni-dir\") pod \"multus-additional-cni-plugins-5qbl5\" (UID: \"6275fc1b-99a7-413c-a0fe-628da6828cd1\") " pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:38.783546 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782958 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6275fc1b-99a7-413c-a0fe-628da6828cd1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5qbl5\" (UID: \"6275fc1b-99a7-413c-a0fe-628da6828cd1\") " pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:38.783546 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.782985 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-etc-systemd\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.783546 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783016 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pk2b\" (UniqueName: \"kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b\") pod \"network-check-target-lzqb8\" (UID: \"e59c8379-7774-4209-bda5-cd8c13a1e464\") " pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:50:38.783546 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783036 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6275fc1b-99a7-413c-a0fe-628da6828cd1-cnibin\") pod \"multus-additional-cni-plugins-5qbl5\" (UID: \"6275fc1b-99a7-413c-a0fe-628da6828cd1\") " pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:38.783546 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783068 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6275fc1b-99a7-413c-a0fe-628da6828cd1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5qbl5\" (UID: \"6275fc1b-99a7-413c-a0fe-628da6828cd1\") " pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:38.783546 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783092 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/92dbe1ae-0e8e-4772-9c97-be594f062d76-host\") pod \"node-ca-rr4fp\" (UID: \"92dbe1ae-0e8e-4772-9c97-be594f062d76\") " pod="openshift-image-registry/node-ca-rr4fp" Sep 08 16:50:38.783546 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783109 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-host-var-lib-kubelet\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.784360 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783131 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-multus-daemon-config\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.784360 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783149 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-host-run-multus-certs\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.784360 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783165 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq5ct\" (UniqueName: \"kubernetes.io/projected/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-kube-api-access-pq5ct\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.784360 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783189 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-var-lib-openvswitch\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.784360 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783206 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.784360 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783220 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf53ac63-f864-402b-bf12-45b289e7e13d-ovn-node-metrics-cert\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.784360 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783240 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a6d1c7fb-9d63-4859-8ac1-46819d815405-agent-certs\") pod \"konnectivity-agent-2ccgf\" (UID: \"a6d1c7fb-9d63-4859-8ac1-46819d815405\") " pod="kube-system/konnectivity-agent-2ccgf" Sep 08 16:50:38.784360 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783272 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-cnibin\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.784360 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783290 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-host-var-lib-cni-bin\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.784360 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783304 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf53ac63-f864-402b-bf12-45b289e7e13d-env-overrides\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.784360 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783320 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6275fc1b-99a7-413c-a0fe-628da6828cd1-cni-binary-copy\") pod \"multus-additional-cni-plugins-5qbl5\" (UID: \"6275fc1b-99a7-413c-a0fe-628da6828cd1\") " pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:38.784360 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783341 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a6d1c7fb-9d63-4859-8ac1-46819d815405-konnectivity-ca\") pod \"konnectivity-agent-2ccgf\" (UID: \"a6d1c7fb-9d63-4859-8ac1-46819d815405\") " pod="kube-system/konnectivity-agent-2ccgf" Sep 08 16:50:38.784360 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783356 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-lib-modules\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.784360 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783369 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15b7eb50-d4dc-4794-9448-164cf91390a9-tmp\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.784360 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783383 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-host-var-lib-cni-multus\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.784360 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783410 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9xsb\" (UniqueName: \"kubernetes.io/projected/1da361fa-6e95-4a71-bac4-e4e5a2c6d5ef-kube-api-access-p9xsb\") pod \"node-resolver-2fv8p\" (UID: \"1da361fa-6e95-4a71-bac4-e4e5a2c6d5ef\") " pod="openshift-dns/node-resolver-2fv8p" Sep 08 16:50:38.784871 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783445 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-host-kubelet\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.784871 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783492 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-host-slash\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.784871 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783516 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-run-systemd\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.784871 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783545 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-run-openvswitch\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.784871 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783564 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-run-ovn\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.784871 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783582 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rxll\" (UniqueName: \"kubernetes.io/projected/bf53ac63-f864-402b-bf12-45b289e7e13d-kube-api-access-4rxll\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.784871 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783617 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwtvt\" (UniqueName: \"kubernetes.io/projected/6275fc1b-99a7-413c-a0fe-628da6828cd1-kube-api-access-cwtvt\") pod \"multus-additional-cni-plugins-5qbl5\" (UID: \"6275fc1b-99a7-413c-a0fe-628da6828cd1\") " pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:38.784871 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783635 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9abcfc03-c9b3-4457-95d8-cc2686ff907a-iptables-alerter-script\") pod \"iptables-alerter-bbtgc\" (UID: \"9abcfc03-c9b3-4457-95d8-cc2686ff907a\") " pod="openshift-network-operator/iptables-alerter-bbtgc" Sep 08 16:50:38.784871 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783653 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58rqt\" (UniqueName: \"kubernetes.io/projected/92dbe1ae-0e8e-4772-9c97-be594f062d76-kube-api-access-58rqt\") pod \"node-ca-rr4fp\" (UID: \"92dbe1ae-0e8e-4772-9c97-be594f062d76\") " pod="openshift-image-registry/node-ca-rr4fp" Sep 08 16:50:38.784871 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783667 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-etc-sysctl-conf\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.784871 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783680 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-etc-kubernetes\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.784871 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783778 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-run\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.784871 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783797 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-multus-socket-dir-parent\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.784871 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783825 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9abcfc03-c9b3-4457-95d8-cc2686ff907a-host-slash\") pod \"iptables-alerter-bbtgc\" (UID: \"9abcfc03-c9b3-4457-95d8-cc2686ff907a\") " pod="openshift-network-operator/iptables-alerter-bbtgc" Sep 08 16:50:38.784871 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783899 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-host-run-ovn-kubernetes\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.784871 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783915 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/192b1a00-b435-4793-98d5-d51cf7b036d1-dbus\") pod \"global-pull-secret-syncer-jpx5h\" (UID: \"192b1a00-b435-4793-98d5-d51cf7b036d1\") " pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:38.785368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783931 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret\") pod \"global-pull-secret-syncer-jpx5h\" (UID: \"192b1a00-b435-4793-98d5-d51cf7b036d1\") " pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:38.785368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783954 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9abcfc03-c9b3-4457-95d8-cc2686ff907a-host-slash\") pod \"iptables-alerter-bbtgc\" (UID: \"9abcfc03-c9b3-4457-95d8-cc2686ff907a\") " pod="openshift-network-operator/iptables-alerter-bbtgc" Sep 08 16:50:38.785368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.783952 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-multus-cni-dir\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.785368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.784251 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9abcfc03-c9b3-4457-95d8-cc2686ff907a-iptables-alerter-script\") pod \"iptables-alerter-bbtgc\" (UID: \"9abcfc03-c9b3-4457-95d8-cc2686ff907a\") " pod="openshift-network-operator/iptables-alerter-bbtgc" Sep 08 16:50:38.788084 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.788065 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 08 16:50:38.791598 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.791572 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lqll\" (UniqueName: \"kubernetes.io/projected/e88b3e9c-6397-4627-a317-933f2576b451-kube-api-access-5lqll\") pod \"aws-ebs-csi-driver-node-jms58\" (UID: \"e88b3e9c-6397-4627-a317-933f2576b451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:38.791686 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.791640 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9xsb\" (UniqueName: \"kubernetes.io/projected/1da361fa-6e95-4a71-bac4-e4e5a2c6d5ef-kube-api-access-p9xsb\") pod \"node-resolver-2fv8p\" (UID: \"1da361fa-6e95-4a71-bac4-e4e5a2c6d5ef\") " pod="openshift-dns/node-resolver-2fv8p" Sep 08 16:50:38.791727 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.791685 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxq29\" (UniqueName: \"kubernetes.io/projected/9abcfc03-c9b3-4457-95d8-cc2686ff907a-kube-api-access-nxq29\") pod \"iptables-alerter-bbtgc\" (UID: \"9abcfc03-c9b3-4457-95d8-cc2686ff907a\") " pod="openshift-network-operator/iptables-alerter-bbtgc" Sep 08 16:50:38.884418 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.884396 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf53ac63-f864-402b-bf12-45b289e7e13d-ovnkube-script-lib\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.884418 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.884422 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/15b7eb50-d4dc-4794-9448-164cf91390a9-etc-tuned\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.884585 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.884452 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n56p\" (UniqueName: \"kubernetes.io/projected/5d001365-79a1-4fec-bcf8-3abf7309f1ff-kube-api-access-9n56p\") pod \"network-metrics-daemon-qbb9j\" (UID: \"5d001365-79a1-4fec-bcf8-3abf7309f1ff\") " pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:50:38.884585 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.884491 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/192b1a00-b435-4793-98d5-d51cf7b036d1-kubelet-config\") pod \"global-pull-secret-syncer-jpx5h\" (UID: \"192b1a00-b435-4793-98d5-d51cf7b036d1\") " pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:38.884585 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.884558 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/192b1a00-b435-4793-98d5-d51cf7b036d1-kubelet-config\") pod \"global-pull-secret-syncer-jpx5h\" (UID: \"192b1a00-b435-4793-98d5-d51cf7b036d1\") " pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:38.884703 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.884670 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6275fc1b-99a7-413c-a0fe-628da6828cd1-system-cni-dir\") pod \"multus-additional-cni-plugins-5qbl5\" (UID: \"6275fc1b-99a7-413c-a0fe-628da6828cd1\") " pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:38.884744 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.884702 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6275fc1b-99a7-413c-a0fe-628da6828cd1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5qbl5\" (UID: \"6275fc1b-99a7-413c-a0fe-628da6828cd1\") " pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:38.884744 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.884737 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-etc-systemd\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.884833 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.884766 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6275fc1b-99a7-413c-a0fe-628da6828cd1-system-cni-dir\") pod \"multus-additional-cni-plugins-5qbl5\" (UID: \"6275fc1b-99a7-413c-a0fe-628da6828cd1\") " pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:38.884833 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.884808 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-etc-systemd\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.884936 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.884840 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pk2b\" (UniqueName: \"kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b\") pod \"network-check-target-lzqb8\" (UID: \"e59c8379-7774-4209-bda5-cd8c13a1e464\") " pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:50:38.884936 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.884869 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6275fc1b-99a7-413c-a0fe-628da6828cd1-cnibin\") pod \"multus-additional-cni-plugins-5qbl5\" (UID: \"6275fc1b-99a7-413c-a0fe-628da6828cd1\") " pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:38.884936 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.884886 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6275fc1b-99a7-413c-a0fe-628da6828cd1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5qbl5\" (UID: \"6275fc1b-99a7-413c-a0fe-628da6828cd1\") " pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:38.884936 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.884902 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/92dbe1ae-0e8e-4772-9c97-be594f062d76-host\") pod \"node-ca-rr4fp\" (UID: \"92dbe1ae-0e8e-4772-9c97-be594f062d76\") " pod="openshift-image-registry/node-ca-rr4fp" Sep 08 16:50:38.884936 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.884923 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-host-var-lib-kubelet\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.885333 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.884968 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/92dbe1ae-0e8e-4772-9c97-be594f062d76-host\") pod \"node-ca-rr4fp\" (UID: \"92dbe1ae-0e8e-4772-9c97-be594f062d76\") " pod="openshift-image-registry/node-ca-rr4fp" Sep 08 16:50:38.885333 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.884971 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6275fc1b-99a7-413c-a0fe-628da6828cd1-cnibin\") pod \"multus-additional-cni-plugins-5qbl5\" (UID: \"6275fc1b-99a7-413c-a0fe-628da6828cd1\") " pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:38.885333 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.884985 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-multus-daemon-config\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.885333 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.884994 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf53ac63-f864-402b-bf12-45b289e7e13d-ovnkube-script-lib\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.885333 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885015 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-host-var-lib-kubelet\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.885333 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885034 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-host-run-multus-certs\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.885333 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885061 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq5ct\" (UniqueName: \"kubernetes.io/projected/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-kube-api-access-pq5ct\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.885333 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885095 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-var-lib-openvswitch\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.885333 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885142 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-host-run-multus-certs\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.885333 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885146 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-var-lib-openvswitch\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.885333 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885258 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.885333 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885291 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf53ac63-f864-402b-bf12-45b289e7e13d-ovn-node-metrics-cert\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.885333 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885293 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6275fc1b-99a7-413c-a0fe-628da6828cd1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5qbl5\" (UID: \"6275fc1b-99a7-413c-a0fe-628da6828cd1\") " pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:38.885333 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885325 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a6d1c7fb-9d63-4859-8ac1-46819d815405-agent-certs\") pod \"konnectivity-agent-2ccgf\" (UID: \"a6d1c7fb-9d63-4859-8ac1-46819d815405\") " pod="kube-system/konnectivity-agent-2ccgf" Sep 08 16:50:38.885333 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885332 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.885333 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885345 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-cnibin\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.886169 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885365 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-host-var-lib-cni-bin\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.886169 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885406 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-host-var-lib-cni-bin\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.886169 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885459 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6275fc1b-99a7-413c-a0fe-628da6828cd1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5qbl5\" (UID: \"6275fc1b-99a7-413c-a0fe-628da6828cd1\") " pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:38.886169 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885483 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf53ac63-f864-402b-bf12-45b289e7e13d-env-overrides\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.886169 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885488 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-cnibin\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.886169 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885520 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6275fc1b-99a7-413c-a0fe-628da6828cd1-cni-binary-copy\") pod \"multus-additional-cni-plugins-5qbl5\" (UID: \"6275fc1b-99a7-413c-a0fe-628da6828cd1\") " pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:38.886169 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885622 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-multus-daemon-config\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.886169 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885633 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a6d1c7fb-9d63-4859-8ac1-46819d815405-konnectivity-ca\") pod \"konnectivity-agent-2ccgf\" (UID: \"a6d1c7fb-9d63-4859-8ac1-46819d815405\") " pod="kube-system/konnectivity-agent-2ccgf" Sep 08 16:50:38.886169 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885661 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-lib-modules\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.886169 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885684 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15b7eb50-d4dc-4794-9448-164cf91390a9-tmp\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.886169 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885755 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-host-var-lib-cni-multus\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.886169 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885784 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-host-kubelet\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.886169 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885833 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-host-slash\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.886169 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885860 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-run-systemd\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.886169 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885886 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-run-openvswitch\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.886169 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885912 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-run-ovn\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.886169 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885924 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-host-kubelet\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.886169 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885940 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rxll\" (UniqueName: \"kubernetes.io/projected/bf53ac63-f864-402b-bf12-45b289e7e13d-kube-api-access-4rxll\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.887135 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885941 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf53ac63-f864-402b-bf12-45b289e7e13d-env-overrides\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.887135 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.885987 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-host-var-lib-cni-multus\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.887135 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886005 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-run-systemd\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.887135 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886017 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwtvt\" (UniqueName: \"kubernetes.io/projected/6275fc1b-99a7-413c-a0fe-628da6828cd1-kube-api-access-cwtvt\") pod \"multus-additional-cni-plugins-5qbl5\" (UID: \"6275fc1b-99a7-413c-a0fe-628da6828cd1\") " pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:38.887135 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886048 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58rqt\" (UniqueName: \"kubernetes.io/projected/92dbe1ae-0e8e-4772-9c97-be594f062d76-kube-api-access-58rqt\") pod \"node-ca-rr4fp\" (UID: \"92dbe1ae-0e8e-4772-9c97-be594f062d76\") " pod="openshift-image-registry/node-ca-rr4fp" Sep 08 16:50:38.887135 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886076 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-etc-sysctl-conf\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.887135 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886100 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-etc-kubernetes\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.887135 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886127 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-run\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.887135 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886151 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-multus-socket-dir-parent\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.887135 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886175 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a6d1c7fb-9d63-4859-8ac1-46819d815405-konnectivity-ca\") pod \"konnectivity-agent-2ccgf\" (UID: \"a6d1c7fb-9d63-4859-8ac1-46819d815405\") " pod="kube-system/konnectivity-agent-2ccgf" Sep 08 16:50:38.887135 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886179 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-host-run-ovn-kubernetes\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.887135 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886213 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-host-run-ovn-kubernetes\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.887135 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886231 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/192b1a00-b435-4793-98d5-d51cf7b036d1-dbus\") pod \"global-pull-secret-syncer-jpx5h\" (UID: \"192b1a00-b435-4793-98d5-d51cf7b036d1\") " pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:38.887135 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886257 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-run-openvswitch\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.887135 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886261 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret\") pod \"global-pull-secret-syncer-jpx5h\" (UID: \"192b1a00-b435-4793-98d5-d51cf7b036d1\") " pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:38.887135 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886296 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-multus-cni-dir\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.887135 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:38.886338 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:50:38.887135 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886328 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-log-socket\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.888067 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886375 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-etc-sysctl-d\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.888067 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886427 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-hostroot\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.888067 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:38.886455 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret podName:192b1a00-b435-4793-98d5-d51cf7b036d1 nodeName:}" failed. No retries permitted until 2026-09-08 16:50:39.386431601 +0000 UTC m=+2.123262715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret") pod "global-pull-secret-syncer-jpx5h" (UID: "192b1a00-b435-4793-98d5-d51cf7b036d1") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:50:38.888067 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886469 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-hostroot\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.888067 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886491 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-host-cni-netd\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.888067 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886539 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/92dbe1ae-0e8e-4772-9c97-be594f062d76-serviceca\") pod \"node-ca-rr4fp\" (UID: \"92dbe1ae-0e8e-4772-9c97-be594f062d76\") " pod="openshift-image-registry/node-ca-rr4fp" Sep 08 16:50:38.888067 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886560 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-run-ovn\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.888067 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886570 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-host-run-netns\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.888067 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886598 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-host\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.888067 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886623 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-multus-cni-dir\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.888067 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886626 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-host-cni-bin\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.888067 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886657 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf53ac63-f864-402b-bf12-45b289e7e13d-ovnkube-config\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.888067 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886666 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-host-cni-bin\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.888067 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886684 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-etc-sysctl-conf\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.888067 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886733 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-var-lib-kubelet\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.888067 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886781 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/192b1a00-b435-4793-98d5-d51cf7b036d1-dbus\") pod \"global-pull-secret-syncer-jpx5h\" (UID: \"192b1a00-b435-4793-98d5-d51cf7b036d1\") " pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:38.888067 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886796 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-etc-sysctl-d\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.888657 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886820 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-log-socket\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.888657 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886888 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-etc-kubernetes\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.888657 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886899 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-host-cni-netd\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.888657 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886933 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-run\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.888657 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886981 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-multus-socket-dir-parent\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.888657 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.886684 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-var-lib-kubelet\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.888657 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887024 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-system-cni-dir\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.888657 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887052 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-etc-modprobe-d\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.888657 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887078 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwtbq\" (UniqueName: \"kubernetes.io/projected/15b7eb50-d4dc-4794-9448-164cf91390a9-kube-api-access-vwtbq\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.888657 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887103 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-etc-kubernetes\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.888657 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887129 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs\") pod \"network-metrics-daemon-qbb9j\" (UID: \"5d001365-79a1-4fec-bcf8-3abf7309f1ff\") " pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:50:38.888657 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887156 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-etc-openvswitch\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.888657 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887183 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6275fc1b-99a7-413c-a0fe-628da6828cd1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5qbl5\" (UID: \"6275fc1b-99a7-413c-a0fe-628da6828cd1\") " pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:38.888657 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887209 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-etc-sysconfig\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.888657 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887237 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-host-run-k8s-cni-cncf-io\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.888657 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887252 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/92dbe1ae-0e8e-4772-9c97-be594f062d76-serviceca\") pod \"node-ca-rr4fp\" (UID: \"92dbe1ae-0e8e-4772-9c97-be594f062d76\") " pod="openshift-image-registry/node-ca-rr4fp" Sep 08 16:50:38.888657 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887263 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-host-run-netns\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.888657 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887290 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-host-run-netns\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.889170 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887289 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-multus-conf-dir\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.889170 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887326 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-host\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.889170 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887332 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf53ac63-f864-402b-bf12-45b289e7e13d-ovnkube-config\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.889170 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887335 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-node-log\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.889170 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887374 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-sys\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.889170 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887376 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-node-log\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.889170 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887394 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-lib-modules\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.889170 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887400 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-cni-binary-copy\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.889170 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887427 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-etc-openvswitch\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.889170 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887434 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-system-cni-dir\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.889170 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887430 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-systemd-units\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.889170 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887539 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-systemd-units\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.889170 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887580 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-etc-kubernetes\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.889170 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887608 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6275fc1b-99a7-413c-a0fe-628da6828cd1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5qbl5\" (UID: \"6275fc1b-99a7-413c-a0fe-628da6828cd1\") " pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:38.889170 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:38.887613 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:50:38.889170 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887626 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-sys\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.889170 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887653 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf53ac63-f864-402b-bf12-45b289e7e13d-host-slash\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.889170 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887687 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf53ac63-f864-402b-bf12-45b289e7e13d-ovn-node-metrics-cert\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.889170 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887688 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/15b7eb50-d4dc-4794-9448-164cf91390a9-etc-tuned\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.889836 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:38.887704 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs podName:5d001365-79a1-4fec-bcf8-3abf7309f1ff nodeName:}" failed. No retries permitted until 2026-09-08 16:50:39.387679281 +0000 UTC m=+2.124510411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs") pod "network-metrics-daemon-qbb9j" (UID: "5d001365-79a1-4fec-bcf8-3abf7309f1ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:50:38.889836 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887709 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-host-run-k8s-cni-cncf-io\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.889836 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887542 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-etc-modprobe-d\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.889836 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887762 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-multus-conf-dir\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.889836 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887779 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/15b7eb50-d4dc-4794-9448-164cf91390a9-etc-sysconfig\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.889836 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.887801 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-host-run-netns\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.889836 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.888039 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6275fc1b-99a7-413c-a0fe-628da6828cd1-cni-binary-copy\") pod \"multus-additional-cni-plugins-5qbl5\" (UID: \"6275fc1b-99a7-413c-a0fe-628da6828cd1\") " pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:38.889836 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.888111 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a6d1c7fb-9d63-4859-8ac1-46819d815405-agent-certs\") pod \"konnectivity-agent-2ccgf\" (UID: \"a6d1c7fb-9d63-4859-8ac1-46819d815405\") " pod="kube-system/konnectivity-agent-2ccgf" Sep 08 16:50:38.889836 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.888153 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-cni-binary-copy\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.889836 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.889636 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15b7eb50-d4dc-4794-9448-164cf91390a9-tmp\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.896266 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.896243 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n56p\" (UniqueName: \"kubernetes.io/projected/5d001365-79a1-4fec-bcf8-3abf7309f1ff-kube-api-access-9n56p\") pod \"network-metrics-daemon-qbb9j\" (UID: \"5d001365-79a1-4fec-bcf8-3abf7309f1ff\") " pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:50:38.897443 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:38.897428 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:50:38.897586 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:38.897447 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:50:38.897586 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:38.897457 2565 projected.go:196] Error preparing data for projected volume kube-api-access-9pk2b for pod openshift-network-diagnostics/network-check-target-lzqb8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:50:38.897586 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:38.897518 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b podName:e59c8379-7774-4209-bda5-cd8c13a1e464 nodeName:}" failed. No retries permitted until 2026-09-08 16:50:39.39748606 +0000 UTC m=+2.134317182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9pk2b" (UniqueName: "kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b") pod "network-check-target-lzqb8" (UID: "e59c8379-7774-4209-bda5-cd8c13a1e464") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:50:38.899804 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.899779 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq5ct\" (UniqueName: \"kubernetes.io/projected/149ee825-8ec8-4b9b-8c8c-c96c5b724b29-kube-api-access-pq5ct\") pod \"multus-9lwh2\" (UID: \"149ee825-8ec8-4b9b-8c8c-c96c5b724b29\") " pod="openshift-multus/multus-9lwh2" Sep 08 16:50:38.901027 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.900999 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rxll\" (UniqueName: \"kubernetes.io/projected/bf53ac63-f864-402b-bf12-45b289e7e13d-kube-api-access-4rxll\") pod \"ovnkube-node-9xxnl\" (UID: \"bf53ac63-f864-402b-bf12-45b289e7e13d\") " pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:38.901176 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.901154 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58rqt\" (UniqueName: \"kubernetes.io/projected/92dbe1ae-0e8e-4772-9c97-be594f062d76-kube-api-access-58rqt\") pod \"node-ca-rr4fp\" (UID: \"92dbe1ae-0e8e-4772-9c97-be594f062d76\") " pod="openshift-image-registry/node-ca-rr4fp" Sep 08 16:50:38.901611 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.901594 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwtvt\" (UniqueName: \"kubernetes.io/projected/6275fc1b-99a7-413c-a0fe-628da6828cd1-kube-api-access-cwtvt\") pod \"multus-additional-cni-plugins-5qbl5\" (UID: \"6275fc1b-99a7-413c-a0fe-628da6828cd1\") " pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:38.902064 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.902047 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwtbq\" (UniqueName: \"kubernetes.io/projected/15b7eb50-d4dc-4794-9448-164cf91390a9-kube-api-access-vwtbq\") pod \"tuned-zdmwv\" (UID: \"15b7eb50-d4dc-4794-9448-164cf91390a9\") " pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:38.985869 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:38.985843 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:50:39.000963 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.000937 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2fv8p" Sep 08 16:50:39.010675 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.010653 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" Sep 08 16:50:39.024481 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.024464 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bbtgc" Sep 08 16:50:39.052283 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.052264 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:50:39.059112 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.059086 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" Sep 08 16:50:39.070306 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.070283 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9lwh2" Sep 08 16:50:39.091750 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.091730 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5qbl5" Sep 08 16:50:39.110772 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.110736 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2ccgf" Sep 08 16:50:39.115646 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.115627 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rr4fp" Sep 08 16:50:39.329177 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:39.329135 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209ac989f5bdb50086e7e268153789b2.slice/crio-ce4076df1771393dba85160a279f3f7da6d66b2b3117f7c089b3ed39065bef1c WatchSource:0}: Error finding container ce4076df1771393dba85160a279f3f7da6d66b2b3117f7c089b3ed39065bef1c: Status 404 returned error can't find the container with id ce4076df1771393dba85160a279f3f7da6d66b2b3117f7c089b3ed39065bef1c Sep 08 16:50:39.334417 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.334397 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 16:50:39.335900 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:39.335878 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01139cbc3b4b1c8ef9cbefa8f1a92b6f.slice/crio-c02e8770b900812f36cfcdf9efa05d077be26a41a31929d267e3555045cdd5ce WatchSource:0}: Error finding container c02e8770b900812f36cfcdf9efa05d077be26a41a31929d267e3555045cdd5ce: Status 404 returned error can't find the container with id c02e8770b900812f36cfcdf9efa05d077be26a41a31929d267e3555045cdd5ce Sep 08 16:50:39.390632 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.390604 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs\") pod \"network-metrics-daemon-qbb9j\" (UID: \"5d001365-79a1-4fec-bcf8-3abf7309f1ff\") " pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:50:39.390732 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.390658 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret\") pod \"global-pull-secret-syncer-jpx5h\" (UID: \"192b1a00-b435-4793-98d5-d51cf7b036d1\") " pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:39.390784 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:39.390749 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:50:39.390832 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:39.390820 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs podName:5d001365-79a1-4fec-bcf8-3abf7309f1ff nodeName:}" failed. No retries permitted until 2026-09-08 16:50:40.390804536 +0000 UTC m=+3.127635632 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs") pod "network-metrics-daemon-qbb9j" (UID: "5d001365-79a1-4fec-bcf8-3abf7309f1ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:50:39.390887 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:39.390753 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:50:39.390887 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:39.390884 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret podName:192b1a00-b435-4793-98d5-d51cf7b036d1 nodeName:}" failed. No retries permitted until 2026-09-08 16:50:40.390871824 +0000 UTC m=+3.127702920 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret") pod "global-pull-secret-syncer-jpx5h" (UID: "192b1a00-b435-4793-98d5-d51cf7b036d1") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:50:39.404679 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:39.404655 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf53ac63_f864_402b_bf12_45b289e7e13d.slice/crio-c0823a99c50b8b1aae05db3a89f7e4c47af1c37bdc3c82f445e55e859587af42 WatchSource:0}: Error finding container c0823a99c50b8b1aae05db3a89f7e4c47af1c37bdc3c82f445e55e859587af42: Status 404 returned error can't find the container with id c0823a99c50b8b1aae05db3a89f7e4c47af1c37bdc3c82f445e55e859587af42 Sep 08 16:50:39.468237 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:39.468212 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b7eb50_d4dc_4794_9448_164cf91390a9.slice/crio-a4f932fe0cbc664552da54b1c12ef6a14cb08e37bd5517df08e913e9d51e1230 WatchSource:0}: Error finding container a4f932fe0cbc664552da54b1c12ef6a14cb08e37bd5517df08e913e9d51e1230: Status 404 returned error can't find the container with id a4f932fe0cbc664552da54b1c12ef6a14cb08e37bd5517df08e913e9d51e1230 Sep 08 16:50:39.491971 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.491942 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pk2b\" (UniqueName: \"kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b\") pod \"network-check-target-lzqb8\" (UID: \"e59c8379-7774-4209-bda5-cd8c13a1e464\") " pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:50:39.492111 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:39.492093 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:50:39.492156 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:39.492119 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:50:39.492156 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:39.492129 2565 projected.go:196] Error preparing data for projected volume kube-api-access-9pk2b for pod openshift-network-diagnostics/network-check-target-lzqb8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:50:39.492225 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:39.492173 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b podName:e59c8379-7774-4209-bda5-cd8c13a1e464 nodeName:}" failed. No retries permitted until 2026-09-08 16:50:40.49215952 +0000 UTC m=+3.228990620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pk2b" (UniqueName: "kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b") pod "network-check-target-lzqb8" (UID: "e59c8379-7774-4209-bda5-cd8c13a1e464") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:50:39.501329 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:39.501163 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92dbe1ae_0e8e_4772_9c97_be594f062d76.slice/crio-bab724e086ef5de1fe472cce250e8208cb805e6d48b3e892858aab66f8202a53 WatchSource:0}: Error finding container bab724e086ef5de1fe472cce250e8208cb805e6d48b3e892858aab66f8202a53: Status 404 returned error can't find the container with id bab724e086ef5de1fe472cce250e8208cb805e6d48b3e892858aab66f8202a53 Sep 08 16:50:39.501690 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:39.501631 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88b3e9c_6397_4627_a317_933f2576b451.slice/crio-3f82bb3a5cdab58174c7985f5b0775c6050e262d4be0e0f26d0f98874354c14a WatchSource:0}: Error finding container 3f82bb3a5cdab58174c7985f5b0775c6050e262d4be0e0f26d0f98874354c14a: Status 404 returned error can't find the container with id 3f82bb3a5cdab58174c7985f5b0775c6050e262d4be0e0f26d0f98874354c14a Sep 08 16:50:39.544738 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:39.544718 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod149ee825_8ec8_4b9b_8c8c_c96c5b724b29.slice/crio-d6d0f390c722ae2423fe2d1c3361e09cf8544e4fc03e49e1792b194e0e29512b WatchSource:0}: Error finding container d6d0f390c722ae2423fe2d1c3361e09cf8544e4fc03e49e1792b194e0e29512b: Status 404 returned error can't find the container with id d6d0f390c722ae2423fe2d1c3361e09cf8544e4fc03e49e1792b194e0e29512b Sep 08 16:50:39.728006 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.727952 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 16:45:39 +0000 UTC" deadline="2028-03-05 03:27:36.386122 +0000 UTC" Sep 08 16:50:39.728006 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.728003 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13042h36m56.658123094s" Sep 08 16:50:39.757252 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.757204 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" event={"ID":"15b7eb50-d4dc-4794-9448-164cf91390a9","Type":"ContainerStarted","Data":"a4f932fe0cbc664552da54b1c12ef6a14cb08e37bd5517df08e913e9d51e1230"} Sep 08 16:50:39.758536 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.758509 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" event={"ID":"bf53ac63-f864-402b-bf12-45b289e7e13d","Type":"ContainerStarted","Data":"c0823a99c50b8b1aae05db3a89f7e4c47af1c37bdc3c82f445e55e859587af42"} Sep 08 16:50:39.759603 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.759581 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-108.ec2.internal" event={"ID":"01139cbc3b4b1c8ef9cbefa8f1a92b6f","Type":"ContainerStarted","Data":"c02e8770b900812f36cfcdf9efa05d077be26a41a31929d267e3555045cdd5ce"} Sep 08 16:50:39.760602 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.760579 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-108.ec2.internal" event={"ID":"209ac989f5bdb50086e7e268153789b2","Type":"ContainerStarted","Data":"ce4076df1771393dba85160a279f3f7da6d66b2b3117f7c089b3ed39065bef1c"} Sep 08 16:50:39.761651 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.761632 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bbtgc" event={"ID":"9abcfc03-c9b3-4457-95d8-cc2686ff907a","Type":"ContainerStarted","Data":"9af90c261934a0f50bd1550d9030a68dca1acdcfd347cc4a717c9d1e174cd2ab"} Sep 08 16:50:39.762712 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.762693 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9lwh2" event={"ID":"149ee825-8ec8-4b9b-8c8c-c96c5b724b29","Type":"ContainerStarted","Data":"d6d0f390c722ae2423fe2d1c3361e09cf8544e4fc03e49e1792b194e0e29512b"} Sep 08 16:50:39.763816 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.763790 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" event={"ID":"e88b3e9c-6397-4627-a317-933f2576b451","Type":"ContainerStarted","Data":"3f82bb3a5cdab58174c7985f5b0775c6050e262d4be0e0f26d0f98874354c14a"} Sep 08 16:50:39.764869 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.764850 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rr4fp" event={"ID":"92dbe1ae-0e8e-4772-9c97-be594f062d76","Type":"ContainerStarted","Data":"bab724e086ef5de1fe472cce250e8208cb805e6d48b3e892858aab66f8202a53"} Sep 08 16:50:39.805254 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:39.805229 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6275fc1b_99a7_413c_a0fe_628da6828cd1.slice/crio-42c69d2a68a3320be5d05b99cd7957e3eaddf5d0157766a8fdb04f3fe81083c3 WatchSource:0}: Error finding container 42c69d2a68a3320be5d05b99cd7957e3eaddf5d0157766a8fdb04f3fe81083c3: Status 404 returned error can't find the container with id 42c69d2a68a3320be5d05b99cd7957e3eaddf5d0157766a8fdb04f3fe81083c3 Sep 08 16:50:39.995677 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:39.995608 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:50:40.088049 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:40.088016 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d1c7fb_9d63_4859_8ac1_46819d815405.slice/crio-6eae105569bc8dab5c4d70dba27e4f8388222d8accf97af445e9cf1d4b7f2f89 WatchSource:0}: Error finding container 6eae105569bc8dab5c4d70dba27e4f8388222d8accf97af445e9cf1d4b7f2f89: Status 404 returned error can't find the container with id 6eae105569bc8dab5c4d70dba27e4f8388222d8accf97af445e9cf1d4b7f2f89 Sep 08 16:50:40.333905 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:40.333827 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:50:40.364182 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:50:40.364112 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da361fa_6e95_4a71_bac4_e4e5a2c6d5ef.slice/crio-c24786934b742e94fd37f4085c2ac931179f025a9dc4b680938df7da2ca66098 WatchSource:0}: Error finding container c24786934b742e94fd37f4085c2ac931179f025a9dc4b680938df7da2ca66098: Status 404 returned error can't find the container with id c24786934b742e94fd37f4085c2ac931179f025a9dc4b680938df7da2ca66098 Sep 08 16:50:40.401022 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:40.400234 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret\") pod \"global-pull-secret-syncer-jpx5h\" (UID: \"192b1a00-b435-4793-98d5-d51cf7b036d1\") " pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:40.401022 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:40.400294 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs\") pod \"network-metrics-daemon-qbb9j\" (UID: \"5d001365-79a1-4fec-bcf8-3abf7309f1ff\") " pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:50:40.401022 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:40.400449 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:50:40.401022 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:40.400529 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs podName:5d001365-79a1-4fec-bcf8-3abf7309f1ff nodeName:}" failed. No retries permitted until 2026-09-08 16:50:42.400509719 +0000 UTC m=+5.137340833 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs") pod "network-metrics-daemon-qbb9j" (UID: "5d001365-79a1-4fec-bcf8-3abf7309f1ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:50:40.401022 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:40.400928 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:50:40.401022 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:40.400981 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret podName:192b1a00-b435-4793-98d5-d51cf7b036d1 nodeName:}" failed. No retries permitted until 2026-09-08 16:50:42.40096464 +0000 UTC m=+5.137795750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret") pod "global-pull-secret-syncer-jpx5h" (UID: "192b1a00-b435-4793-98d5-d51cf7b036d1") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:50:40.500992 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:40.500954 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pk2b\" (UniqueName: \"kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b\") pod \"network-check-target-lzqb8\" (UID: \"e59c8379-7774-4209-bda5-cd8c13a1e464\") " pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:50:40.501173 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:40.501143 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:50:40.501173 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:40.501169 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:50:40.501293 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:40.501183 2565 projected.go:196] Error preparing data for projected volume kube-api-access-9pk2b for pod openshift-network-diagnostics/network-check-target-lzqb8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:50:40.501293 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:40.501244 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b podName:e59c8379-7774-4209-bda5-cd8c13a1e464 nodeName:}" failed. No retries permitted until 2026-09-08 16:50:42.501224114 +0000 UTC m=+5.238055228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pk2b" (UniqueName: "kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b") pod "network-check-target-lzqb8" (UID: "e59c8379-7774-4209-bda5-cd8c13a1e464") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:50:40.728791 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:40.728746 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 16:45:39 +0000 UTC" deadline="2028-02-23 09:59:22.796517833 +0000 UTC" Sep 08 16:50:40.728791 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:40.728789 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12785h8m42.067734225s" Sep 08 16:50:40.754245 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:40.754217 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:50:40.754396 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:40.754347 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzqb8" podUID="e59c8379-7774-4209-bda5-cd8c13a1e464" Sep 08 16:50:40.754794 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:40.754774 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:40.754894 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:40.754876 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jpx5h" podUID="192b1a00-b435-4793-98d5-d51cf7b036d1" Sep 08 16:50:40.754967 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:40.754940 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:50:40.755056 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:40.755036 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb9j" podUID="5d001365-79a1-4fec-bcf8-3abf7309f1ff" Sep 08 16:50:40.779364 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:40.779334 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2fv8p" event={"ID":"1da361fa-6e95-4a71-bac4-e4e5a2c6d5ef","Type":"ContainerStarted","Data":"c24786934b742e94fd37f4085c2ac931179f025a9dc4b680938df7da2ca66098"} Sep 08 16:50:40.783607 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:40.783583 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2ccgf" event={"ID":"a6d1c7fb-9d63-4859-8ac1-46819d815405","Type":"ContainerStarted","Data":"6eae105569bc8dab5c4d70dba27e4f8388222d8accf97af445e9cf1d4b7f2f89"} Sep 08 16:50:40.801236 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:40.797374 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qbl5" event={"ID":"6275fc1b-99a7-413c-a0fe-628da6828cd1","Type":"ContainerStarted","Data":"42c69d2a68a3320be5d05b99cd7957e3eaddf5d0157766a8fdb04f3fe81083c3"} Sep 08 16:50:41.214577 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:41.214543 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:50:42.417611 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:42.417524 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret\") pod \"global-pull-secret-syncer-jpx5h\" (UID: \"192b1a00-b435-4793-98d5-d51cf7b036d1\") " pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:42.417611 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:42.417585 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs\") pod \"network-metrics-daemon-qbb9j\" (UID: \"5d001365-79a1-4fec-bcf8-3abf7309f1ff\") " pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:50:42.418125 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:42.417714 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:50:42.418125 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:42.417780 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs podName:5d001365-79a1-4fec-bcf8-3abf7309f1ff nodeName:}" failed. No retries permitted until 2026-09-08 16:50:46.417759216 +0000 UTC m=+9.154590336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs") pod "network-metrics-daemon-qbb9j" (UID: "5d001365-79a1-4fec-bcf8-3abf7309f1ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:50:42.418307 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:42.418248 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:50:42.418393 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:42.418310 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret podName:192b1a00-b435-4793-98d5-d51cf7b036d1 nodeName:}" failed. No retries permitted until 2026-09-08 16:50:46.41829279 +0000 UTC m=+9.155123887 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret") pod "global-pull-secret-syncer-jpx5h" (UID: "192b1a00-b435-4793-98d5-d51cf7b036d1") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:50:42.519073 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:42.518451 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pk2b\" (UniqueName: \"kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b\") pod \"network-check-target-lzqb8\" (UID: \"e59c8379-7774-4209-bda5-cd8c13a1e464\") " pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:50:42.519073 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:42.518658 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:50:42.519073 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:42.518681 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:50:42.519073 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:42.518694 2565 projected.go:196] Error preparing data for projected volume kube-api-access-9pk2b for pod openshift-network-diagnostics/network-check-target-lzqb8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:50:42.519073 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:42.518756 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b podName:e59c8379-7774-4209-bda5-cd8c13a1e464 nodeName:}" failed. No retries permitted until 2026-09-08 16:50:46.518736949 +0000 UTC m=+9.255568050 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pk2b" (UniqueName: "kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b") pod "network-check-target-lzqb8" (UID: "e59c8379-7774-4209-bda5-cd8c13a1e464") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:50:42.754210 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:42.754175 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:50:42.754386 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:42.754313 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzqb8" podUID="e59c8379-7774-4209-bda5-cd8c13a1e464" Sep 08 16:50:42.754763 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:42.754740 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:42.754871 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:42.754851 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jpx5h" podUID="192b1a00-b435-4793-98d5-d51cf7b036d1" Sep 08 16:50:42.754936 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:42.754921 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:50:42.755036 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:42.755017 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb9j" podUID="5d001365-79a1-4fec-bcf8-3abf7309f1ff" Sep 08 16:50:44.754156 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:44.754121 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:50:44.754693 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:44.754277 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb9j" podUID="5d001365-79a1-4fec-bcf8-3abf7309f1ff" Sep 08 16:50:44.754756 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:44.754692 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:50:44.754814 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:44.754799 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzqb8" podUID="e59c8379-7774-4209-bda5-cd8c13a1e464" Sep 08 16:50:44.754881 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:44.754842 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:44.754936 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:44.754921 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jpx5h" podUID="192b1a00-b435-4793-98d5-d51cf7b036d1" Sep 08 16:50:46.454949 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:46.454917 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs\") pod \"network-metrics-daemon-qbb9j\" (UID: \"5d001365-79a1-4fec-bcf8-3abf7309f1ff\") " pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:50:46.455416 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:46.454991 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret\") pod \"global-pull-secret-syncer-jpx5h\" (UID: \"192b1a00-b435-4793-98d5-d51cf7b036d1\") " pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:46.455416 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:46.455114 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:50:46.455416 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:46.455176 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret podName:192b1a00-b435-4793-98d5-d51cf7b036d1 nodeName:}" failed. No retries permitted until 2026-09-08 16:50:54.455158455 +0000 UTC m=+17.191989551 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret") pod "global-pull-secret-syncer-jpx5h" (UID: "192b1a00-b435-4793-98d5-d51cf7b036d1") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:50:46.455416 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:46.455263 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:50:46.455416 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:46.455329 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs podName:5d001365-79a1-4fec-bcf8-3abf7309f1ff nodeName:}" failed. No retries permitted until 2026-09-08 16:50:54.455310374 +0000 UTC m=+17.192141488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs") pod "network-metrics-daemon-qbb9j" (UID: "5d001365-79a1-4fec-bcf8-3abf7309f1ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:50:46.555752 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:46.555675 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pk2b\" (UniqueName: \"kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b\") pod \"network-check-target-lzqb8\" (UID: \"e59c8379-7774-4209-bda5-cd8c13a1e464\") " pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:50:46.555899 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:46.555837 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:50:46.555899 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:46.555856 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:50:46.555899 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:46.555868 2565 projected.go:196] Error preparing data for projected volume kube-api-access-9pk2b for pod openshift-network-diagnostics/network-check-target-lzqb8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:50:46.556027 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:46.555922 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b podName:e59c8379-7774-4209-bda5-cd8c13a1e464 nodeName:}" failed. No retries permitted until 2026-09-08 16:50:54.555904793 +0000 UTC m=+17.292735906 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pk2b" (UniqueName: "kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b") pod "network-check-target-lzqb8" (UID: "e59c8379-7774-4209-bda5-cd8c13a1e464") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:50:46.753909 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:46.753878 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:46.753909 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:46.753904 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:50:46.754144 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:46.753878 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:50:46.754144 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:46.754013 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jpx5h" podUID="192b1a00-b435-4793-98d5-d51cf7b036d1" Sep 08 16:50:46.754144 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:46.754130 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb9j" podUID="5d001365-79a1-4fec-bcf8-3abf7309f1ff" Sep 08 16:50:46.754288 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:46.754208 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzqb8" podUID="e59c8379-7774-4209-bda5-cd8c13a1e464" Sep 08 16:50:48.754335 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:48.753566 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:50:48.754335 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:48.753688 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzqb8" podUID="e59c8379-7774-4209-bda5-cd8c13a1e464" Sep 08 16:50:48.754335 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:48.754063 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:48.754335 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:48.754165 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jpx5h" podUID="192b1a00-b435-4793-98d5-d51cf7b036d1" Sep 08 16:50:48.754335 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:48.754216 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:50:48.754335 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:48.754289 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb9j" podUID="5d001365-79a1-4fec-bcf8-3abf7309f1ff" Sep 08 16:50:50.754212 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:50.754123 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:50.754212 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:50.754161 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:50:50.754212 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:50.754161 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:50:50.754725 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:50.754259 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jpx5h" podUID="192b1a00-b435-4793-98d5-d51cf7b036d1" Sep 08 16:50:50.754725 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:50.754365 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzqb8" podUID="e59c8379-7774-4209-bda5-cd8c13a1e464" Sep 08 16:50:50.754725 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:50.754481 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb9j" podUID="5d001365-79a1-4fec-bcf8-3abf7309f1ff" Sep 08 16:50:52.753676 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:52.753647 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:50:52.754014 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:52.753732 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:52.754014 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:52.753740 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzqb8" podUID="e59c8379-7774-4209-bda5-cd8c13a1e464" Sep 08 16:50:52.754014 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:52.753819 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jpx5h" podUID="192b1a00-b435-4793-98d5-d51cf7b036d1" Sep 08 16:50:52.754014 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:52.753855 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:50:52.754014 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:52.753911 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb9j" podUID="5d001365-79a1-4fec-bcf8-3abf7309f1ff" Sep 08 16:50:54.515736 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:54.515694 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret\") pod \"global-pull-secret-syncer-jpx5h\" (UID: \"192b1a00-b435-4793-98d5-d51cf7b036d1\") " pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:54.516163 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:54.515757 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs\") pod \"network-metrics-daemon-qbb9j\" (UID: \"5d001365-79a1-4fec-bcf8-3abf7309f1ff\") " pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:50:54.516163 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:54.515855 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:50:54.516163 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:54.515864 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:50:54.516163 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:54.515908 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs podName:5d001365-79a1-4fec-bcf8-3abf7309f1ff nodeName:}" failed. No retries permitted until 2026-09-08 16:51:10.515894751 +0000 UTC m=+33.252725846 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs") pod "network-metrics-daemon-qbb9j" (UID: "5d001365-79a1-4fec-bcf8-3abf7309f1ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:50:54.516163 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:54.515932 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret podName:192b1a00-b435-4793-98d5-d51cf7b036d1 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:10.515913552 +0000 UTC m=+33.252744667 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret") pod "global-pull-secret-syncer-jpx5h" (UID: "192b1a00-b435-4793-98d5-d51cf7b036d1") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:50:54.616233 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:54.616197 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pk2b\" (UniqueName: \"kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b\") pod \"network-check-target-lzqb8\" (UID: \"e59c8379-7774-4209-bda5-cd8c13a1e464\") " pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:50:54.616400 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:54.616368 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:50:54.616400 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:54.616391 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:50:54.616400 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:54.616402 2565 projected.go:196] Error preparing data for projected volume kube-api-access-9pk2b for pod openshift-network-diagnostics/network-check-target-lzqb8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:50:54.616578 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:54.616476 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b podName:e59c8379-7774-4209-bda5-cd8c13a1e464 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:10.616457723 +0000 UTC m=+33.353288841 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pk2b" (UniqueName: "kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b") pod "network-check-target-lzqb8" (UID: "e59c8379-7774-4209-bda5-cd8c13a1e464") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:50:54.753987 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:54.753950 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:50:54.754161 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:54.754053 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:54.754161 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:54.754052 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzqb8" podUID="e59c8379-7774-4209-bda5-cd8c13a1e464" Sep 08 16:50:54.754161 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:54.754056 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:50:54.754847 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:54.754167 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jpx5h" podUID="192b1a00-b435-4793-98d5-d51cf7b036d1" Sep 08 16:50:54.754847 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:54.754234 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb9j" podUID="5d001365-79a1-4fec-bcf8-3abf7309f1ff" Sep 08 16:50:56.754140 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:56.754106 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:50:56.754140 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:56.754129 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:50:56.754687 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:56.754103 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:56.754687 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:56.754268 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb9j" podUID="5d001365-79a1-4fec-bcf8-3abf7309f1ff" Sep 08 16:50:56.754687 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:56.754315 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jpx5h" podUID="192b1a00-b435-4793-98d5-d51cf7b036d1" Sep 08 16:50:56.754687 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:56.754370 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzqb8" podUID="e59c8379-7774-4209-bda5-cd8c13a1e464" Sep 08 16:50:58.753493 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:58.753452 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:50:58.753960 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:58.753467 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:50:58.753960 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:58.753468 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:50:58.753960 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:58.753584 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzqb8" podUID="e59c8379-7774-4209-bda5-cd8c13a1e464" Sep 08 16:50:58.753960 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:58.753683 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jpx5h" podUID="192b1a00-b435-4793-98d5-d51cf7b036d1" Sep 08 16:50:58.753960 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:50:58.753774 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb9j" podUID="5d001365-79a1-4fec-bcf8-3abf7309f1ff" Sep 08 16:50:59.839513 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:59.839228 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9lwh2" event={"ID":"149ee825-8ec8-4b9b-8c8c-c96c5b724b29","Type":"ContainerStarted","Data":"1d2699ca04c4c9ab250f6cacfdb660ed03c6b49dbb8dadf48b102052e936b518"} Sep 08 16:50:59.841250 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:59.841226 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" event={"ID":"15b7eb50-d4dc-4794-9448-164cf91390a9","Type":"ContainerStarted","Data":"e41f2246fabc0495e068f038ba78effb62a5fc6661044e37bb3c13bac7acec35"} Sep 08 16:50:59.843960 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:59.843936 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xxnl_bf53ac63-f864-402b-bf12-45b289e7e13d/ovn-acl-logging/0.log" Sep 08 16:50:59.844334 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:59.844316 2565 generic.go:356] "Generic (PLEG): container finished" podID="bf53ac63-f864-402b-bf12-45b289e7e13d" containerID="eee495c9a1aac80f28ab8c06c9aa3b215e36dea759d5f0bb076c3cf9cf006ce5" exitCode=1 Sep 08 16:50:59.844446 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:59.844385 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" event={"ID":"bf53ac63-f864-402b-bf12-45b289e7e13d","Type":"ContainerStarted","Data":"06f195ca8e2402d0456940c7669a4c5ed37f18dac8c3f6a60c5a68483578d349"} Sep 08 16:50:59.844522 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:59.844484 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" event={"ID":"bf53ac63-f864-402b-bf12-45b289e7e13d","Type":"ContainerStarted","Data":"e687386a800016ec94581175111c4f1798d8b830c0a784cfe16ed75ca1a95be5"} Sep 08 16:50:59.844577 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:59.844542 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" event={"ID":"bf53ac63-f864-402b-bf12-45b289e7e13d","Type":"ContainerDied","Data":"eee495c9a1aac80f28ab8c06c9aa3b215e36dea759d5f0bb076c3cf9cf006ce5"} Sep 08 16:50:59.844627 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:59.844592 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" event={"ID":"bf53ac63-f864-402b-bf12-45b289e7e13d","Type":"ContainerStarted","Data":"d1d129c01d2fff6f285ac5bf989070cb810f8f469b787d365a491071d5dd7c71"} Sep 08 16:50:59.849124 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:59.847364 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-108.ec2.internal" event={"ID":"209ac989f5bdb50086e7e268153789b2","Type":"ContainerStarted","Data":"0e54a9b1d72a975c5a00a80cf176acde626755903d3eea2af81bff1ba0f50b02"} Sep 08 16:50:59.857531 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:59.857337 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9lwh2" podStartSLOduration=1.872670619 podStartE2EDuration="21.857319026s" totalImagesPullingTime="19.984648407s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:50:59.856830844 +0000 UTC m=+22.593661963" watchObservedRunningTime="2026-09-08 16:50:59.857319026 +0000 UTC m=+22.594150143" Sep 08 16:50:59.871497 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:59.871459 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-108.ec2.internal" podStartSLOduration=20.871411252 podStartE2EDuration="20.871411252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:50:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:50:59.871287155 +0000 UTC m=+22.608118273" watchObservedRunningTime="2026-09-08 16:50:59.871411252 +0000 UTC m=+22.608242368" Sep 08 16:50:59.886243 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:50:59.886065 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zdmwv" podStartSLOduration=2.106601711 podStartE2EDuration="21.886047392s" totalImagesPullingTime="19.779445681s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:50:59.885671948 +0000 UTC m=+22.622503066" watchObservedRunningTime="2026-09-08 16:50:59.886047392 +0000 UTC m=+22.622878510" Sep 08 16:51:00.753874 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:00.753855 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:51:00.754007 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:00.753976 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:51:00.754007 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:00.753982 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzqb8" podUID="e59c8379-7774-4209-bda5-cd8c13a1e464" Sep 08 16:51:00.754076 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:00.753981 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:51:00.754076 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:00.754055 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jpx5h" podUID="192b1a00-b435-4793-98d5-d51cf7b036d1" Sep 08 16:51:00.754160 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:00.754143 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb9j" podUID="5d001365-79a1-4fec-bcf8-3abf7309f1ff" Sep 08 16:51:00.851187 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:00.851159 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rr4fp" event={"ID":"92dbe1ae-0e8e-4772-9c97-be594f062d76","Type":"ContainerStarted","Data":"7db6b49d6eb72dca6130b921377d8f1a9bcf375c94d3be18aac73e2da63b1843"} Sep 08 16:51:00.852626 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:00.852604 2565 generic.go:356] "Generic (PLEG): container finished" podID="01139cbc3b4b1c8ef9cbefa8f1a92b6f" containerID="9ab444c0d21430478b914ba0594af8a976640f9434bad6beca74c702de562693" exitCode=0 Sep 08 16:51:00.852731 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:00.852671 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-108.ec2.internal" event={"ID":"01139cbc3b4b1c8ef9cbefa8f1a92b6f","Type":"ContainerDied","Data":"9ab444c0d21430478b914ba0594af8a976640f9434bad6beca74c702de562693"} Sep 08 16:51:00.854087 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:00.854061 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2fv8p" event={"ID":"1da361fa-6e95-4a71-bac4-e4e5a2c6d5ef","Type":"ContainerStarted","Data":"4bcef05638a05571485cf1cdc6432b3963ff8b33a157f2ff33ccf5394d0018e7"} Sep 08 16:51:00.855536 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:00.855512 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2ccgf" event={"ID":"a6d1c7fb-9d63-4859-8ac1-46819d815405","Type":"ContainerStarted","Data":"20407a96f581cc6cfdfec1e1eb95b9aa6c85c5862a169fb639036a64a4fa2318"} Sep 08 16:51:00.856920 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:00.856898 2565 generic.go:356] "Generic (PLEG): container finished" podID="6275fc1b-99a7-413c-a0fe-628da6828cd1" containerID="6778d3500e96dda17367b28deb98ce0a097fb6242ca05ee32491bb6e8aaaeac4" exitCode=0 Sep 08 16:51:00.856996 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:00.856978 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qbl5" event={"ID":"6275fc1b-99a7-413c-a0fe-628da6828cd1","Type":"ContainerDied","Data":"6778d3500e96dda17367b28deb98ce0a097fb6242ca05ee32491bb6e8aaaeac4"} Sep 08 16:51:00.858649 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:00.858628 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" event={"ID":"e88b3e9c-6397-4627-a317-933f2576b451","Type":"ContainerStarted","Data":"90589b27fb5d61258b054401a1d4827e2580f1123025acd6284b407d70e7529d"} Sep 08 16:51:00.861382 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:00.861368 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xxnl_bf53ac63-f864-402b-bf12-45b289e7e13d/ovn-acl-logging/0.log" Sep 08 16:51:00.861805 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:00.861787 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" event={"ID":"bf53ac63-f864-402b-bf12-45b289e7e13d","Type":"ContainerStarted","Data":"ca60960789bbc2b01b83d50ca54c2313afc1141c0e52bb6967b5e2e45c366bea"} Sep 08 16:51:00.861880 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:00.861838 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" event={"ID":"bf53ac63-f864-402b-bf12-45b289e7e13d","Type":"ContainerStarted","Data":"c3b2c2d23cc1e872c3ac313c49cd51d9c1dcd5d6a49b162d2dbd93f0f0ae7dd4"} Sep 08 16:51:00.883791 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:00.883745 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2ccgf" podStartSLOduration=3.754504539 podStartE2EDuration="22.883729458s" totalImagesPullingTime="19.129224919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:00.883454305 +0000 UTC m=+23.620285420" watchObservedRunningTime="2026-09-08 16:51:00.883729458 +0000 UTC m=+23.620560575" Sep 08 16:51:00.883935 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:00.883913 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rr4fp" podStartSLOduration=3.133675555 podStartE2EDuration="22.883908585s" totalImagesPullingTime="19.75023303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:00.870707877 +0000 UTC m=+23.607538993" watchObservedRunningTime="2026-09-08 16:51:00.883908585 +0000 UTC m=+23.620739702" Sep 08 16:51:00.914463 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:00.914397 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2fv8p" podStartSLOduration=4.027562038 podStartE2EDuration="22.914383219s" totalImagesPullingTime="18.886821181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:00.914118205 +0000 UTC m=+23.650949321" watchObservedRunningTime="2026-09-08 16:51:00.914383219 +0000 UTC m=+23.651214335" Sep 08 16:51:01.652447 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:01.652402 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 08 16:51:01.865360 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:01.865275 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bbtgc" event={"ID":"9abcfc03-c9b3-4457-95d8-cc2686ff907a","Type":"ContainerStarted","Data":"046eae0f5fc0a81650a31f5afb52f215095ea6eaf7fb896ffa5c78cd76b687cd"} Sep 08 16:51:01.867010 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:01.866990 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-108.ec2.internal" event={"ID":"01139cbc3b4b1c8ef9cbefa8f1a92b6f","Type":"ContainerStarted","Data":"54102db2f17ffabf4d5e0262bf1d64cea4e11be4c568ea753882dcfecb12331f"} Sep 08 16:51:01.868738 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:01.868711 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" event={"ID":"e88b3e9c-6397-4627-a317-933f2576b451","Type":"ContainerStarted","Data":"b507cb9b9795a891759545e5c7b745948e9e63505d75664b887621ad098e8bb4"} Sep 08 16:51:01.883860 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:01.883786 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T16:51:01.652440412Z","UUID":"33374081-6632-4089-af15-fecc75fdfc4a","Handler":null,"Name":"","Endpoint":""} Sep 08 16:51:01.885312 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:01.885292 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 08 16:51:01.885439 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:01.885326 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 08 16:51:01.899576 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:01.899532 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bbtgc" podStartSLOduration=4.342112773 podStartE2EDuration="23.899518018s" totalImagesPullingTime="19.557405245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:01.881703672 +0000 UTC m=+24.618534788" watchObservedRunningTime="2026-09-08 16:51:01.899518018 +0000 UTC m=+24.636349165" Sep 08 16:51:01.899823 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:01.899804 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-108.ec2.internal" podStartSLOduration=22.899799581 podStartE2EDuration="22.899799581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:50:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:01.89937751 +0000 UTC m=+24.636208625" watchObservedRunningTime="2026-09-08 16:51:01.899799581 +0000 UTC m=+24.636630697" Sep 08 16:51:02.754172 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:02.753937 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:51:02.754351 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:02.753966 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:51:02.754351 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:02.754254 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jpx5h" podUID="192b1a00-b435-4793-98d5-d51cf7b036d1" Sep 08 16:51:02.754520 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:02.754354 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb9j" podUID="5d001365-79a1-4fec-bcf8-3abf7309f1ff" Sep 08 16:51:02.754520 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:02.753966 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:51:02.754520 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:02.754492 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzqb8" podUID="e59c8379-7774-4209-bda5-cd8c13a1e464" Sep 08 16:51:02.873769 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:02.873744 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xxnl_bf53ac63-f864-402b-bf12-45b289e7e13d/ovn-acl-logging/0.log" Sep 08 16:51:02.874246 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:02.874207 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" event={"ID":"bf53ac63-f864-402b-bf12-45b289e7e13d","Type":"ContainerStarted","Data":"2028a59363fe13d99365621e78d7d185a114f8416403469ff5b9bba86b598ffc"} Sep 08 16:51:02.960368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:02.960339 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2ccgf" Sep 08 16:51:02.961046 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:02.961026 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2ccgf" Sep 08 16:51:03.877467 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:03.877418 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2ccgf" Sep 08 16:51:03.877969 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:03.877938 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2ccgf" Sep 08 16:51:04.754146 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:04.754105 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:51:04.754312 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:04.754156 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:51:04.754312 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:04.754156 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:51:04.754312 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:04.754218 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jpx5h" podUID="192b1a00-b435-4793-98d5-d51cf7b036d1" Sep 08 16:51:04.754510 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:04.754328 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzqb8" podUID="e59c8379-7774-4209-bda5-cd8c13a1e464" Sep 08 16:51:04.754510 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:04.754459 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb9j" podUID="5d001365-79a1-4fec-bcf8-3abf7309f1ff" Sep 08 16:51:04.882026 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:04.881952 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" event={"ID":"e88b3e9c-6397-4627-a317-933f2576b451","Type":"ContainerStarted","Data":"d242e74c772810ab91320da8f392575cf9a8dcb2d8443cad550878379a3675ec"} Sep 08 16:51:04.885836 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:04.885810 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xxnl_bf53ac63-f864-402b-bf12-45b289e7e13d/ovn-acl-logging/0.log" Sep 08 16:51:04.886243 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:04.886206 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" event={"ID":"bf53ac63-f864-402b-bf12-45b289e7e13d","Type":"ContainerStarted","Data":"ec67936cba1e84d9f3a0a20a3673585928ba3b1dbae70843f2a99ff07d1e0835"} Sep 08 16:51:04.886552 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:04.886491 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:51:04.886648 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:04.886559 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:51:04.886648 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:04.886614 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:51:04.886761 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:04.886673 2565 scope.go:117] "RemoveContainer" containerID="eee495c9a1aac80f28ab8c06c9aa3b215e36dea759d5f0bb076c3cf9cf006ce5" Sep 08 16:51:04.899616 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:04.899365 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jms58" podStartSLOduration=3.289255266 podStartE2EDuration="26.899345627s" totalImagesPullingTime="23.610090361s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:50:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:04.899113877 +0000 UTC m=+27.635944993" watchObservedRunningTime="2026-09-08 16:51:04.899345627 +0000 UTC m=+27.636176747" Sep 08 16:51:04.902719 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:04.902655 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:51:04.903720 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:04.903702 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:51:06.174627 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:06.174593 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qbb9j"] Sep 08 16:51:06.175010 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:06.174755 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:51:06.175010 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:06.174880 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb9j" podUID="5d001365-79a1-4fec-bcf8-3abf7309f1ff" Sep 08 16:51:06.178498 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:06.178468 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lzqb8"] Sep 08 16:51:06.178613 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:06.178514 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jpx5h"] Sep 08 16:51:06.178613 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:06.178581 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:51:06.178742 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:06.178669 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jpx5h" podUID="192b1a00-b435-4793-98d5-d51cf7b036d1" Sep 08 16:51:06.178742 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:06.178737 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:51:06.178825 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:06.178807 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzqb8" podUID="e59c8379-7774-4209-bda5-cd8c13a1e464" Sep 08 16:51:06.893735 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:06.893661 2565 generic.go:356] "Generic (PLEG): container finished" podID="6275fc1b-99a7-413c-a0fe-628da6828cd1" containerID="0542338998f6f8bc97a1c81aabe9d27c9e5b526045b4fe57cd0eff6dafcadf5a" exitCode=0 Sep 08 16:51:06.893865 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:06.893743 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qbl5" event={"ID":"6275fc1b-99a7-413c-a0fe-628da6828cd1","Type":"ContainerDied","Data":"0542338998f6f8bc97a1c81aabe9d27c9e5b526045b4fe57cd0eff6dafcadf5a"} Sep 08 16:51:06.897768 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:06.897752 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xxnl_bf53ac63-f864-402b-bf12-45b289e7e13d/ovn-acl-logging/0.log" Sep 08 16:51:06.898091 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:06.898073 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" event={"ID":"bf53ac63-f864-402b-bf12-45b289e7e13d","Type":"ContainerStarted","Data":"62bd156fa393c7521ffe32cb0f215de3189c58ffba7e7adf1f9816cee76849ac"} Sep 08 16:51:06.940648 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:06.940606 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" podStartSLOduration=9.090776864 podStartE2EDuration="28.940592989s" totalImagesPullingTime="19.849816125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:06.940570067 +0000 UTC m=+29.677401185" watchObservedRunningTime="2026-09-08 16:51:06.940592989 +0000 UTC m=+29.677424101" Sep 08 16:51:07.754628 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:07.754604 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:51:07.755451 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:07.754700 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jpx5h" podUID="192b1a00-b435-4793-98d5-d51cf7b036d1" Sep 08 16:51:07.755451 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:07.754786 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:51:07.755451 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:07.754874 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:51:07.755451 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:07.754921 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzqb8" podUID="e59c8379-7774-4209-bda5-cd8c13a1e464" Sep 08 16:51:07.755451 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:07.754918 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb9j" podUID="5d001365-79a1-4fec-bcf8-3abf7309f1ff" Sep 08 16:51:08.904565 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:08.904531 2565 generic.go:356] "Generic (PLEG): container finished" podID="6275fc1b-99a7-413c-a0fe-628da6828cd1" containerID="2ccee6b874c2754c0ba59b78e29a2bfd731c28fb2a1394897478cbf15c8035da" exitCode=0 Sep 08 16:51:08.905052 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:08.904577 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qbl5" event={"ID":"6275fc1b-99a7-413c-a0fe-628da6828cd1","Type":"ContainerDied","Data":"2ccee6b874c2754c0ba59b78e29a2bfd731c28fb2a1394897478cbf15c8035da"} Sep 08 16:51:09.753830 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:09.753652 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:51:09.754009 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:09.753671 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:51:09.754009 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:09.753900 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jpx5h" podUID="192b1a00-b435-4793-98d5-d51cf7b036d1" Sep 08 16:51:09.754009 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:09.753670 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:51:09.754139 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:09.753975 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb9j" podUID="5d001365-79a1-4fec-bcf8-3abf7309f1ff" Sep 08 16:51:09.754139 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:09.754063 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzqb8" podUID="e59c8379-7774-4209-bda5-cd8c13a1e464" Sep 08 16:51:10.526122 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:10.526087 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs\") pod \"network-metrics-daemon-qbb9j\" (UID: \"5d001365-79a1-4fec-bcf8-3abf7309f1ff\") " pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:51:10.526496 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:10.526194 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret\") pod \"global-pull-secret-syncer-jpx5h\" (UID: \"192b1a00-b435-4793-98d5-d51cf7b036d1\") " pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:51:10.526496 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:10.526219 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:51:10.526496 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:10.526280 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs podName:5d001365-79a1-4fec-bcf8-3abf7309f1ff nodeName:}" failed. No retries permitted until 2026-09-08 16:51:42.52626387 +0000 UTC m=+65.263095021 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs") pod "network-metrics-daemon-qbb9j" (UID: "5d001365-79a1-4fec-bcf8-3abf7309f1ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:51:10.526496 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:10.526294 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:51:10.526496 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:10.526343 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret podName:192b1a00-b435-4793-98d5-d51cf7b036d1 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:42.526327913 +0000 UTC m=+65.263159021 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret") pod "global-pull-secret-syncer-jpx5h" (UID: "192b1a00-b435-4793-98d5-d51cf7b036d1") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:51:10.627264 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:10.627238 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pk2b\" (UniqueName: \"kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b\") pod \"network-check-target-lzqb8\" (UID: \"e59c8379-7774-4209-bda5-cd8c13a1e464\") " pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:51:10.627407 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:10.627361 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:51:10.627407 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:10.627376 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:51:10.627407 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:10.627395 2565 projected.go:196] Error preparing data for projected volume kube-api-access-9pk2b for pod openshift-network-diagnostics/network-check-target-lzqb8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:51:10.627541 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:10.627457 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b podName:e59c8379-7774-4209-bda5-cd8c13a1e464 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:42.627443984 +0000 UTC m=+65.364275078 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pk2b" (UniqueName: "kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b") pod "network-check-target-lzqb8" (UID: "e59c8379-7774-4209-bda5-cd8c13a1e464") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:51:11.753533 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:11.753502 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:51:11.753964 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:11.753500 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:51:11.753964 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:11.753608 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb9j" podUID="5d001365-79a1-4fec-bcf8-3abf7309f1ff" Sep 08 16:51:11.753964 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:11.753622 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:51:11.753964 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:11.753706 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jpx5h" podUID="192b1a00-b435-4793-98d5-d51cf7b036d1" Sep 08 16:51:11.753964 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:11.753763 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzqb8" podUID="e59c8379-7774-4209-bda5-cd8c13a1e464" Sep 08 16:51:11.913378 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:11.913349 2565 generic.go:356] "Generic (PLEG): container finished" podID="6275fc1b-99a7-413c-a0fe-628da6828cd1" containerID="587d9f8d19a1fbb421cd277f9ffe83af84f02480fe2cc41fa4016c1fee2a40b7" exitCode=0 Sep 08 16:51:11.913549 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:11.913403 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qbl5" event={"ID":"6275fc1b-99a7-413c-a0fe-628da6828cd1","Type":"ContainerDied","Data":"587d9f8d19a1fbb421cd277f9ffe83af84f02480fe2cc41fa4016c1fee2a40b7"} Sep 08 16:51:12.118156 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.118089 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-108.ec2.internal" event="NodeReady" Sep 08 16:51:12.118281 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.118196 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 08 16:51:12.163105 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.163075 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69c8ff9cf7-wh72j"] Sep 08 16:51:12.203211 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.203186 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-69lb7"] Sep 08 16:51:12.203356 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.203320 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.205689 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.205662 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 08 16:51:12.205812 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.205724 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-f9rmv\"" Sep 08 16:51:12.205999 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.205985 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 08 16:51:12.207366 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.207348 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 08 16:51:12.210583 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.210564 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 08 16:51:12.224416 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.224278 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ms7mp"] Sep 08 16:51:12.226700 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.225934 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-69lb7" Sep 08 16:51:12.228963 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.228945 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 08 16:51:12.229041 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.228949 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 08 16:51:12.229284 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.229268 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zpm9l\"" Sep 08 16:51:12.246202 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.246174 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69c8ff9cf7-wh72j"] Sep 08 16:51:12.246305 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.246213 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ms7mp"] Sep 08 16:51:12.246305 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.246235 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-69lb7"] Sep 08 16:51:12.246305 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.246275 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ms7mp" Sep 08 16:51:12.248483 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.248462 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 08 16:51:12.248578 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.248509 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 08 16:51:12.248578 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.248461 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qdttq\"" Sep 08 16:51:12.248682 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.248461 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 08 16:51:12.341137 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.341106 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls\") pod \"dns-default-69lb7\" (UID: \"8ed0286d-2b34-44d3-95da-2d99e0772a69\") " pod="openshift-dns/dns-default-69lb7" Sep 08 16:51:12.341260 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.341145 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k592f\" (UniqueName: \"kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-kube-api-access-k592f\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.341260 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.341186 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-ca-trust-extracted\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.341260 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.341223 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ed0286d-2b34-44d3-95da-2d99e0772a69-tmp-dir\") pod \"dns-default-69lb7\" (UID: \"8ed0286d-2b34-44d3-95da-2d99e0772a69\") " pod="openshift-dns/dns-default-69lb7" Sep 08 16:51:12.341260 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.341249 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b46e7a48-1c3e-4499-b640-c3bd623f857d-cert\") pod \"ingress-canary-ms7mp\" (UID: \"b46e7a48-1c3e-4499-b640-c3bd623f857d\") " pod="openshift-ingress-canary/ingress-canary-ms7mp" Sep 08 16:51:12.341445 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.341266 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkvlb\" (UniqueName: \"kubernetes.io/projected/b46e7a48-1c3e-4499-b640-c3bd623f857d-kube-api-access-hkvlb\") pod \"ingress-canary-ms7mp\" (UID: \"b46e7a48-1c3e-4499-b640-c3bd623f857d\") " pod="openshift-ingress-canary/ingress-canary-ms7mp" Sep 08 16:51:12.341445 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.341283 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-image-registry-private-configuration\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.341445 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.341365 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-bound-sa-token\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.341445 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.341400 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.341445 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.341419 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp58l\" (UniqueName: \"kubernetes.io/projected/8ed0286d-2b34-44d3-95da-2d99e0772a69-kube-api-access-hp58l\") pod \"dns-default-69lb7\" (UID: \"8ed0286d-2b34-44d3-95da-2d99e0772a69\") " pod="openshift-dns/dns-default-69lb7" Sep 08 16:51:12.341659 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.341482 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ed0286d-2b34-44d3-95da-2d99e0772a69-config-volume\") pod \"dns-default-69lb7\" (UID: \"8ed0286d-2b34-44d3-95da-2d99e0772a69\") " pod="openshift-dns/dns-default-69lb7" Sep 08 16:51:12.341659 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.341515 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-trusted-ca\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.341659 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.341534 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-certificates\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.341659 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.341553 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-installation-pull-secrets\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.344190 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.344162 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:12.432203 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.432177 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:12.442155 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.442134 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ed0286d-2b34-44d3-95da-2d99e0772a69-tmp-dir\") pod \"dns-default-69lb7\" (UID: \"8ed0286d-2b34-44d3-95da-2d99e0772a69\") " pod="openshift-dns/dns-default-69lb7" Sep 08 16:51:12.442228 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.442161 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b46e7a48-1c3e-4499-b640-c3bd623f857d-cert\") pod \"ingress-canary-ms7mp\" (UID: \"b46e7a48-1c3e-4499-b640-c3bd623f857d\") " pod="openshift-ingress-canary/ingress-canary-ms7mp" Sep 08 16:51:12.442228 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.442177 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkvlb\" (UniqueName: \"kubernetes.io/projected/b46e7a48-1c3e-4499-b640-c3bd623f857d-kube-api-access-hkvlb\") pod \"ingress-canary-ms7mp\" (UID: \"b46e7a48-1c3e-4499-b640-c3bd623f857d\") " pod="openshift-ingress-canary/ingress-canary-ms7mp" Sep 08 16:51:12.442228 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.442199 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-image-registry-private-configuration\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.442357 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.442249 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-bound-sa-token\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.442357 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.442281 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.442357 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.442304 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp58l\" (UniqueName: \"kubernetes.io/projected/8ed0286d-2b34-44d3-95da-2d99e0772a69-kube-api-access-hp58l\") pod \"dns-default-69lb7\" (UID: \"8ed0286d-2b34-44d3-95da-2d99e0772a69\") " pod="openshift-dns/dns-default-69lb7" Sep 08 16:51:12.442357 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.442328 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ed0286d-2b34-44d3-95da-2d99e0772a69-config-volume\") pod \"dns-default-69lb7\" (UID: \"8ed0286d-2b34-44d3-95da-2d99e0772a69\") " pod="openshift-dns/dns-default-69lb7" Sep 08 16:51:12.442571 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:12.442414 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 16:51:12.442571 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:12.442449 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8ff9cf7-wh72j: secret "image-registry-tls" not found Sep 08 16:51:12.442571 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:12.442511 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls podName:4ec136d4-4af2-4a3b-9462-c21f278d4b4a nodeName:}" failed. No retries permitted until 2026-09-08 16:51:12.942494193 +0000 UTC m=+35.679325305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls") pod "image-registry-69c8ff9cf7-wh72j" (UID: "4ec136d4-4af2-4a3b-9462-c21f278d4b4a") : secret "image-registry-tls" not found Sep 08 16:51:12.442719 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.442639 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-trusted-ca\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.442719 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.442666 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-certificates\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.442719 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.442684 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-installation-pull-secrets\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.442719 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.442686 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ed0286d-2b34-44d3-95da-2d99e0772a69-tmp-dir\") pod \"dns-default-69lb7\" (UID: \"8ed0286d-2b34-44d3-95da-2d99e0772a69\") " pod="openshift-dns/dns-default-69lb7" Sep 08 16:51:12.442719 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.442717 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls\") pod \"dns-default-69lb7\" (UID: \"8ed0286d-2b34-44d3-95da-2d99e0772a69\") " pod="openshift-dns/dns-default-69lb7" Sep 08 16:51:12.442958 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.442736 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k592f\" (UniqueName: \"kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-kube-api-access-k592f\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.442958 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.442761 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-ca-trust-extracted\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.442958 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:12.442830 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 16:51:12.442958 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:12.442888 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls podName:8ed0286d-2b34-44d3-95da-2d99e0772a69 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:12.942871251 +0000 UTC m=+35.679702382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls") pod "dns-default-69lb7" (UID: "8ed0286d-2b34-44d3-95da-2d99e0772a69") : secret "dns-default-metrics-tls" not found Sep 08 16:51:12.443142 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.442980 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ed0286d-2b34-44d3-95da-2d99e0772a69-config-volume\") pod \"dns-default-69lb7\" (UID: \"8ed0286d-2b34-44d3-95da-2d99e0772a69\") " pod="openshift-dns/dns-default-69lb7" Sep 08 16:51:12.443142 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.443090 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-ca-trust-extracted\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.443545 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.443496 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-certificates\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.443806 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.443787 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-trusted-ca\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.446332 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.446314 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-installation-pull-secrets\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.446438 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.446333 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-image-registry-private-configuration\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.446438 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.446338 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b46e7a48-1c3e-4499-b640-c3bd623f857d-cert\") pod \"ingress-canary-ms7mp\" (UID: \"b46e7a48-1c3e-4499-b640-c3bd623f857d\") " pod="openshift-ingress-canary/ingress-canary-ms7mp" Sep 08 16:51:12.453405 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.453377 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp58l\" (UniqueName: \"kubernetes.io/projected/8ed0286d-2b34-44d3-95da-2d99e0772a69-kube-api-access-hp58l\") pod \"dns-default-69lb7\" (UID: \"8ed0286d-2b34-44d3-95da-2d99e0772a69\") " pod="openshift-dns/dns-default-69lb7" Sep 08 16:51:12.453500 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.453482 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-bound-sa-token\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.453607 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.453584 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k592f\" (UniqueName: \"kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-kube-api-access-k592f\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.453739 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.453718 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkvlb\" (UniqueName: \"kubernetes.io/projected/b46e7a48-1c3e-4499-b640-c3bd623f857d-kube-api-access-hkvlb\") pod \"ingress-canary-ms7mp\" (UID: \"b46e7a48-1c3e-4499-b640-c3bd623f857d\") " pod="openshift-ingress-canary/ingress-canary-ms7mp" Sep 08 16:51:12.554494 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.554468 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ms7mp" Sep 08 16:51:12.872678 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.872375 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ms7mp"] Sep 08 16:51:12.878104 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:51:12.878073 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46e7a48_1c3e_4499_b640_c3bd623f857d.slice/crio-ffd2da0e9387f120cd3817f19bffe808c8f86415f9b695eec3a1f2935d84b237 WatchSource:0}: Error finding container ffd2da0e9387f120cd3817f19bffe808c8f86415f9b695eec3a1f2935d84b237: Status 404 returned error can't find the container with id ffd2da0e9387f120cd3817f19bffe808c8f86415f9b695eec3a1f2935d84b237 Sep 08 16:51:12.917214 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.917184 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ms7mp" event={"ID":"b46e7a48-1c3e-4499-b640-c3bd623f857d","Type":"ContainerStarted","Data":"ffd2da0e9387f120cd3817f19bffe808c8f86415f9b695eec3a1f2935d84b237"} Sep 08 16:51:12.946639 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.946609 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:12.946763 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:12.946679 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls\") pod \"dns-default-69lb7\" (UID: \"8ed0286d-2b34-44d3-95da-2d99e0772a69\") " pod="openshift-dns/dns-default-69lb7" Sep 08 16:51:12.946806 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:12.946782 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 16:51:12.946872 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:12.946843 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls podName:8ed0286d-2b34-44d3-95da-2d99e0772a69 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:13.946822675 +0000 UTC m=+36.683653790 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls") pod "dns-default-69lb7" (UID: "8ed0286d-2b34-44d3-95da-2d99e0772a69") : secret "dns-default-metrics-tls" not found Sep 08 16:51:12.946872 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:12.946781 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 16:51:12.947023 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:12.946888 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8ff9cf7-wh72j: secret "image-registry-tls" not found Sep 08 16:51:12.947023 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:12.946943 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls podName:4ec136d4-4af2-4a3b-9462-c21f278d4b4a nodeName:}" failed. No retries permitted until 2026-09-08 16:51:13.946926941 +0000 UTC m=+36.683758050 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls") pod "image-registry-69c8ff9cf7-wh72j" (UID: "4ec136d4-4af2-4a3b-9462-c21f278d4b4a") : secret "image-registry-tls" not found Sep 08 16:51:13.753791 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:13.753736 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:51:13.754076 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:13.753736 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:51:13.754076 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:13.753890 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:51:13.757107 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:13.757084 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wrtqz\"" Sep 08 16:51:13.757211 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:13.757120 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 16:51:13.757211 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:13.757161 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 16:51:13.757313 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:13.757257 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 16:51:13.757368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:13.757344 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8582r\"" Sep 08 16:51:13.757419 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:13.757394 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 16:51:13.954720 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:13.954686 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:13.955184 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:13.954851 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls\") pod \"dns-default-69lb7\" (UID: \"8ed0286d-2b34-44d3-95da-2d99e0772a69\") " pod="openshift-dns/dns-default-69lb7" Sep 08 16:51:13.955184 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:13.954864 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 16:51:13.955184 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:13.954884 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8ff9cf7-wh72j: secret "image-registry-tls" not found Sep 08 16:51:13.955184 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:13.954943 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 16:51:13.955184 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:13.954951 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls podName:4ec136d4-4af2-4a3b-9462-c21f278d4b4a nodeName:}" failed. No retries permitted until 2026-09-08 16:51:15.954931164 +0000 UTC m=+38.691762262 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls") pod "image-registry-69c8ff9cf7-wh72j" (UID: "4ec136d4-4af2-4a3b-9462-c21f278d4b4a") : secret "image-registry-tls" not found Sep 08 16:51:13.955184 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:13.954981 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls podName:8ed0286d-2b34-44d3-95da-2d99e0772a69 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:15.954969558 +0000 UTC m=+38.691800665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls") pod "dns-default-69lb7" (UID: "8ed0286d-2b34-44d3-95da-2d99e0772a69") : secret "dns-default-metrics-tls" not found Sep 08 16:51:15.973189 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:15.973154 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls\") pod \"dns-default-69lb7\" (UID: \"8ed0286d-2b34-44d3-95da-2d99e0772a69\") " pod="openshift-dns/dns-default-69lb7" Sep 08 16:51:15.973833 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:15.973233 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:15.973833 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:15.973314 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 16:51:15.973833 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:15.973327 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 16:51:15.973833 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:15.973339 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8ff9cf7-wh72j: secret "image-registry-tls" not found Sep 08 16:51:15.973833 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:15.973391 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls podName:4ec136d4-4af2-4a3b-9462-c21f278d4b4a nodeName:}" failed. No retries permitted until 2026-09-08 16:51:19.973373855 +0000 UTC m=+42.710204949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls") pod "image-registry-69c8ff9cf7-wh72j" (UID: "4ec136d4-4af2-4a3b-9462-c21f278d4b4a") : secret "image-registry-tls" not found Sep 08 16:51:15.973833 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:15.973409 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls podName:8ed0286d-2b34-44d3-95da-2d99e0772a69 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:19.973400002 +0000 UTC m=+42.710231098 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls") pod "dns-default-69lb7" (UID: "8ed0286d-2b34-44d3-95da-2d99e0772a69") : secret "dns-default-metrics-tls" not found Sep 08 16:51:17.749930 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:17.749902 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:17.849556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:17.849516 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:17.933005 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:17.932958 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qbl5" event={"ID":"6275fc1b-99a7-413c-a0fe-628da6828cd1","Type":"ContainerStarted","Data":"38f916c8981583a0374dece7b520826cba28d563b3c4d7d8e9145e2921ff0046"} Sep 08 16:51:17.937462 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:17.937365 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ms7mp" event={"ID":"b46e7a48-1c3e-4499-b640-c3bd623f857d","Type":"ContainerStarted","Data":"e456d3900826961d3fef5176bf9c48fb8d754d4e8a4d60a9190f6ca497841a9d"} Sep 08 16:51:17.970717 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:17.970671 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ms7mp" podStartSLOduration=1.154132498 podStartE2EDuration="5.97065506s" totalImagesPullingTime="4.816522562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:17.969628106 +0000 UTC m=+40.706459224" watchObservedRunningTime="2026-09-08 16:51:17.97065506 +0000 UTC m=+40.707486170" Sep 08 16:51:18.942407 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:18.942363 2565 generic.go:356] "Generic (PLEG): container finished" podID="6275fc1b-99a7-413c-a0fe-628da6828cd1" containerID="38f916c8981583a0374dece7b520826cba28d563b3c4d7d8e9145e2921ff0046" exitCode=0 Sep 08 16:51:18.942798 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:18.942452 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qbl5" event={"ID":"6275fc1b-99a7-413c-a0fe-628da6828cd1","Type":"ContainerDied","Data":"38f916c8981583a0374dece7b520826cba28d563b3c4d7d8e9145e2921ff0046"} Sep 08 16:51:19.419172 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:19.419138 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:19.519926 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:19.519890 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:19.619924 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:19.619895 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:19.720446 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:19.720357 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:19.948114 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:19.948086 2565 generic.go:356] "Generic (PLEG): container finished" podID="6275fc1b-99a7-413c-a0fe-628da6828cd1" containerID="c5a930ca357cd3b616c3ebedb02a68ea0c6d1ca56c3f54243d7e22201d3bdb91" exitCode=0 Sep 08 16:51:19.948114 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:19.948137 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qbl5" event={"ID":"6275fc1b-99a7-413c-a0fe-628da6828cd1","Type":"ContainerDied","Data":"c5a930ca357cd3b616c3ebedb02a68ea0c6d1ca56c3f54243d7e22201d3bdb91"} Sep 08 16:51:20.005095 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:20.005073 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls\") pod \"dns-default-69lb7\" (UID: \"8ed0286d-2b34-44d3-95da-2d99e0772a69\") " pod="openshift-dns/dns-default-69lb7" Sep 08 16:51:20.005213 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:20.005142 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:20.005278 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:20.005233 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 16:51:20.005278 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:20.005243 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 16:51:20.005278 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:20.005258 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8ff9cf7-wh72j: secret "image-registry-tls" not found Sep 08 16:51:20.005406 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:20.005309 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls podName:4ec136d4-4af2-4a3b-9462-c21f278d4b4a nodeName:}" failed. No retries permitted until 2026-09-08 16:51:28.00529173 +0000 UTC m=+50.742122844 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls") pod "image-registry-69c8ff9cf7-wh72j" (UID: "4ec136d4-4af2-4a3b-9462-c21f278d4b4a") : secret "image-registry-tls" not found Sep 08 16:51:20.005406 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:20.005327 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls podName:8ed0286d-2b34-44d3-95da-2d99e0772a69 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:28.005318728 +0000 UTC m=+50.742149823 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls") pod "dns-default-69lb7" (UID: "8ed0286d-2b34-44d3-95da-2d99e0772a69") : secret "dns-default-metrics-tls" not found Sep 08 16:51:20.953957 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:20.953920 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qbl5" event={"ID":"6275fc1b-99a7-413c-a0fe-628da6828cd1","Type":"ContainerStarted","Data":"b5b499d802b64f0104bd1a87a23949beb4508ddc887b5817cd6e6e343dc21b91"} Sep 08 16:51:20.975015 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:20.974963 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5qbl5" podStartSLOduration=8.627269734 podStartE2EDuration="42.974947247s" totalImagesPullingTime="34.347677513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:50:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:20.973560474 +0000 UTC m=+43.710391589" watchObservedRunningTime="2026-09-08 16:51:20.974947247 +0000 UTC m=+43.711778363" Sep 08 16:51:21.099799 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:21.099764 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:21.200617 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:21.200580 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:21.379528 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:21.379456 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:25.175759 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.175726 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-r87s8"] Sep 08 16:51:25.207284 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.207258 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2b5lx"] Sep 08 16:51:25.207517 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.207383 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-r87s8" Sep 08 16:51:25.211558 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.211531 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-s89nt\"" Sep 08 16:51:25.221208 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.221190 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-64p5d"] Sep 08 16:51:25.221311 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.221298 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2b5lx" Sep 08 16:51:25.225060 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.225039 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:51:25.225163 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.225126 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 08 16:51:25.225227 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.225160 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-qg77n\"" Sep 08 16:51:25.243278 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.243256 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j5fcl"] Sep 08 16:51:25.243379 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.243363 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-64p5d" Sep 08 16:51:25.245981 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.245963 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:51:25.246179 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.246163 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 08 16:51:25.246282 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.246263 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mvzn8\"" Sep 08 16:51:25.246386 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.246369 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 08 16:51:25.246597 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.246573 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 08 16:51:25.261540 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.261517 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-8xjm5"] Sep 08 16:51:25.261647 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.261629 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j5fcl" Sep 08 16:51:25.264869 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.264850 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 08 16:51:25.265515 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.265498 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cpgpg\"" Sep 08 16:51:25.265614 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.265502 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:51:25.265958 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.265939 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 08 16:51:25.281702 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.281681 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2b5lx"] Sep 08 16:51:25.281783 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.281707 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-r87s8"] Sep 08 16:51:25.281783 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.281721 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-8xjm5"] Sep 08 16:51:25.281783 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.281735 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j5fcl"] Sep 08 16:51:25.281783 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.281762 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-64p5d"] Sep 08 16:51:25.281993 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.281847 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-8xjm5" Sep 08 16:51:25.282911 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.282893 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg"] Sep 08 16:51:25.286725 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.286708 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 08 16:51:25.287160 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.287077 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:51:25.287865 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.287300 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 08 16:51:25.287865 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.287814 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 08 16:51:25.288949 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.288931 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-72mz5\"" Sep 08 16:51:25.292268 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.292254 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 08 16:51:25.295723 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.295709 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg" Sep 08 16:51:25.304854 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.304831 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 08 16:51:25.304952 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.304890 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 08 16:51:25.306043 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.306025 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 08 16:51:25.306043 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.306028 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-sc6wc\"" Sep 08 16:51:25.306219 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.306100 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 08 16:51:25.317111 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.317088 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-t48r4"] Sep 08 16:51:25.337694 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.337666 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-858549dccb-6vpw7"] Sep 08 16:51:25.339365 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.339341 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t48r4" Sep 08 16:51:25.341822 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.341799 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-47mhn\"" Sep 08 16:51:25.343092 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.343072 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 08 16:51:25.343397 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.343368 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:51:25.343697 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.343677 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 08 16:51:25.343905 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.343810 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 08 16:51:25.344835 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.344812 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mcv7\" (UniqueName: \"kubernetes.io/projected/e827254b-0e20-4391-8339-3f6541535bc9-kube-api-access-2mcv7\") pod \"volume-data-source-validator-695c895bd6-2b5lx\" (UID: \"e827254b-0e20-4391-8339-3f6541535bc9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2b5lx" Sep 08 16:51:25.344919 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.344865 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl89b\" (UniqueName: \"kubernetes.io/projected/38e261a0-1dcc-45a8-b32a-48673c0f9482-kube-api-access-fl89b\") pod \"network-check-source-685456655f-r87s8\" (UID: \"38e261a0-1dcc-45a8-b32a-48673c0f9482\") " pod="openshift-network-diagnostics/network-check-source-685456655f-r87s8" Sep 08 16:51:25.344919 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.344903 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-64p5d\" (UID: \"dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-64p5d" Sep 08 16:51:25.345095 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.344932 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-64p5d\" (UID: \"dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-64p5d" Sep 08 16:51:25.345095 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.345050 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shvx8\" (UniqueName: \"kubernetes.io/projected/dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4-kube-api-access-shvx8\") pod \"kube-storage-version-migrator-operator-8d7dd4675-64p5d\" (UID: \"dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-64p5d" Sep 08 16:51:25.355745 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.355728 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-dld4j"] Sep 08 16:51:25.355847 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.355832 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:25.358530 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.358495 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 08 16:51:25.358530 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.358520 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-r84x4\"" Sep 08 16:51:25.358702 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.358557 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 08 16:51:25.358702 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.358581 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 08 16:51:25.358702 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.358602 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 08 16:51:25.358849 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.358793 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 08 16:51:25.358929 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.358914 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 08 16:51:25.377818 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.377796 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg"] Sep 08 16:51:25.377818 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.377822 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-t48r4"] Sep 08 16:51:25.377929 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.377839 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-858549dccb-6vpw7"] Sep 08 16:51:25.377929 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.377851 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-dld4j"] Sep 08 16:51:25.377929 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.377902 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" Sep 08 16:51:25.382051 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.382031 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 08 16:51:25.382189 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.382141 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 08 16:51:25.382530 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.382490 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 08 16:51:25.382626 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.382541 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 08 16:51:25.382626 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.382591 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ngr74\"" Sep 08 16:51:25.388753 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.388738 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 08 16:51:25.445812 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.445757 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d648a28-280d-4584-a061-704134b80708-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ngvwg\" (UID: \"0d648a28-280d-4584-a061-704134b80708\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg" Sep 08 16:51:25.445812 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.445788 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx4cc\" (UniqueName: \"kubernetes.io/projected/706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5-kube-api-access-dx4cc\") pod \"service-ca-operator-859f94c4b-t48r4\" (UID: \"706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t48r4" Sep 08 16:51:25.445812 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.445809 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-stats-auth\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:25.445965 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.445848 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl89b\" (UniqueName: \"kubernetes.io/projected/38e261a0-1dcc-45a8-b32a-48673c0f9482-kube-api-access-fl89b\") pod \"network-check-source-685456655f-r87s8\" (UID: \"38e261a0-1dcc-45a8-b32a-48673c0f9482\") " pod="openshift-network-diagnostics/network-check-source-685456655f-r87s8" Sep 08 16:51:25.446000 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.445962 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-default-certificate\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:25.446000 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.445994 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb26j\" (UniqueName: \"kubernetes.io/projected/0d648a28-280d-4584-a061-704134b80708-kube-api-access-pb26j\") pod \"cluster-monitoring-operator-6cdcc96965-ngvwg\" (UID: \"0d648a28-280d-4584-a061-704134b80708\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg" Sep 08 16:51:25.446097 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.446016 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5-config\") pod \"service-ca-operator-859f94c4b-t48r4\" (UID: \"706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t48r4" Sep 08 16:51:25.446097 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.446051 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shvx8\" (UniqueName: \"kubernetes.io/projected/dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4-kube-api-access-shvx8\") pod \"kube-storage-version-migrator-operator-8d7dd4675-64p5d\" (UID: \"dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-64p5d" Sep 08 16:51:25.446158 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.446132 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc58g\" (UniqueName: \"kubernetes.io/projected/0c66fe27-30fe-4051-9707-acf165ca731f-kube-api-access-zc58g\") pod \"cluster-samples-operator-7cdcfcf76c-j5fcl\" (UID: \"0c66fe27-30fe-4051-9707-acf165ca731f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j5fcl" Sep 08 16:51:25.446190 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.446177 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5977d6b9-f03a-40f2-bdab-9f07d1065c2d-serving-cert\") pod \"console-operator-85b454874b-8xjm5\" (UID: \"5977d6b9-f03a-40f2-bdab-9f07d1065c2d\") " pod="openshift-console-operator/console-operator-85b454874b-8xjm5" Sep 08 16:51:25.446222 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.446204 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5-serving-cert\") pod \"service-ca-operator-859f94c4b-t48r4\" (UID: \"706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t48r4" Sep 08 16:51:25.446254 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.446226 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mcv7\" (UniqueName: \"kubernetes.io/projected/e827254b-0e20-4391-8339-3f6541535bc9-kube-api-access-2mcv7\") pod \"volume-data-source-validator-695c895bd6-2b5lx\" (UID: \"e827254b-0e20-4391-8339-3f6541535bc9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2b5lx" Sep 08 16:51:25.446254 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.446243 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xnv9\" (UniqueName: \"kubernetes.io/projected/ec6cae4f-377a-4717-acb8-63166637113a-kube-api-access-5xnv9\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:25.446323 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.446272 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-64p5d\" (UID: \"dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-64p5d" Sep 08 16:51:25.446323 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.446298 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-64p5d\" (UID: \"dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-64p5d" Sep 08 16:51:25.446439 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.446402 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5977d6b9-f03a-40f2-bdab-9f07d1065c2d-config\") pod \"console-operator-85b454874b-8xjm5\" (UID: \"5977d6b9-f03a-40f2-bdab-9f07d1065c2d\") " pod="openshift-console-operator/console-operator-85b454874b-8xjm5" Sep 08 16:51:25.446474 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.446453 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c66fe27-30fe-4051-9707-acf165ca731f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j5fcl\" (UID: \"0c66fe27-30fe-4051-9707-acf165ca731f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j5fcl" Sep 08 16:51:25.446515 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.446475 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5977d6b9-f03a-40f2-bdab-9f07d1065c2d-trusted-ca\") pod \"console-operator-85b454874b-8xjm5\" (UID: \"5977d6b9-f03a-40f2-bdab-9f07d1065c2d\") " pod="openshift-console-operator/console-operator-85b454874b-8xjm5" Sep 08 16:51:25.446515 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.446500 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec6cae4f-377a-4717-acb8-63166637113a-service-ca-bundle\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:25.446592 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.446515 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-metrics-certs\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:25.446592 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.446548 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt7dt\" (UniqueName: \"kubernetes.io/projected/5977d6b9-f03a-40f2-bdab-9f07d1065c2d-kube-api-access-dt7dt\") pod \"console-operator-85b454874b-8xjm5\" (UID: \"5977d6b9-f03a-40f2-bdab-9f07d1065c2d\") " pod="openshift-console-operator/console-operator-85b454874b-8xjm5" Sep 08 16:51:25.446592 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.446579 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0d648a28-280d-4584-a061-704134b80708-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-ngvwg\" (UID: \"0d648a28-280d-4584-a061-704134b80708\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg" Sep 08 16:51:25.446839 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.446809 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-64p5d\" (UID: \"dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-64p5d" Sep 08 16:51:25.449218 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.449193 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-64p5d\" (UID: \"dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-64p5d" Sep 08 16:51:25.455719 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.455683 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shvx8\" (UniqueName: \"kubernetes.io/projected/dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4-kube-api-access-shvx8\") pod \"kube-storage-version-migrator-operator-8d7dd4675-64p5d\" (UID: \"dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-64p5d" Sep 08 16:51:25.455830 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.455811 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl89b\" (UniqueName: \"kubernetes.io/projected/38e261a0-1dcc-45a8-b32a-48673c0f9482-kube-api-access-fl89b\") pod \"network-check-source-685456655f-r87s8\" (UID: \"38e261a0-1dcc-45a8-b32a-48673c0f9482\") " pod="openshift-network-diagnostics/network-check-source-685456655f-r87s8" Sep 08 16:51:25.455876 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.455811 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mcv7\" (UniqueName: \"kubernetes.io/projected/e827254b-0e20-4391-8339-3f6541535bc9-kube-api-access-2mcv7\") pod \"volume-data-source-validator-695c895bd6-2b5lx\" (UID: \"e827254b-0e20-4391-8339-3f6541535bc9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2b5lx" Sep 08 16:51:25.519077 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.519035 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-r87s8" Sep 08 16:51:25.529178 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.529154 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2b5lx" Sep 08 16:51:25.547431 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.547400 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-default-certificate\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:25.547534 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.547495 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c1c8b5c6-12f4-451f-93ac-7e1878b355be-snapshots\") pod \"insights-operator-76bcd6ccf5-dld4j\" (UID: \"c1c8b5c6-12f4-451f-93ac-7e1878b355be\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" Sep 08 16:51:25.547629 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.547608 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1c8b5c6-12f4-451f-93ac-7e1878b355be-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-dld4j\" (UID: \"c1c8b5c6-12f4-451f-93ac-7e1878b355be\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" Sep 08 16:51:25.547702 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.547646 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb26j\" (UniqueName: \"kubernetes.io/projected/0d648a28-280d-4584-a061-704134b80708-kube-api-access-pb26j\") pod \"cluster-monitoring-operator-6cdcc96965-ngvwg\" (UID: \"0d648a28-280d-4584-a061-704134b80708\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg" Sep 08 16:51:25.547702 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.547664 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnztf\" (UniqueName: \"kubernetes.io/projected/c1c8b5c6-12f4-451f-93ac-7e1878b355be-kube-api-access-mnztf\") pod \"insights-operator-76bcd6ccf5-dld4j\" (UID: \"c1c8b5c6-12f4-451f-93ac-7e1878b355be\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" Sep 08 16:51:25.547801 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.547712 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5-config\") pod \"service-ca-operator-859f94c4b-t48r4\" (UID: \"706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t48r4" Sep 08 16:51:25.547801 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.547752 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1c8b5c6-12f4-451f-93ac-7e1878b355be-serving-cert\") pod \"insights-operator-76bcd6ccf5-dld4j\" (UID: \"c1c8b5c6-12f4-451f-93ac-7e1878b355be\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" Sep 08 16:51:25.547801 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.547777 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc58g\" (UniqueName: \"kubernetes.io/projected/0c66fe27-30fe-4051-9707-acf165ca731f-kube-api-access-zc58g\") pod \"cluster-samples-operator-7cdcfcf76c-j5fcl\" (UID: \"0c66fe27-30fe-4051-9707-acf165ca731f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j5fcl" Sep 08 16:51:25.547948 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.547816 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5977d6b9-f03a-40f2-bdab-9f07d1065c2d-serving-cert\") pod \"console-operator-85b454874b-8xjm5\" (UID: \"5977d6b9-f03a-40f2-bdab-9f07d1065c2d\") " pod="openshift-console-operator/console-operator-85b454874b-8xjm5" Sep 08 16:51:25.547948 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.547837 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5-serving-cert\") pod \"service-ca-operator-859f94c4b-t48r4\" (UID: \"706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t48r4" Sep 08 16:51:25.547948 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.547862 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xnv9\" (UniqueName: \"kubernetes.io/projected/ec6cae4f-377a-4717-acb8-63166637113a-kube-api-access-5xnv9\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:25.547948 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.547909 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5977d6b9-f03a-40f2-bdab-9f07d1065c2d-config\") pod \"console-operator-85b454874b-8xjm5\" (UID: \"5977d6b9-f03a-40f2-bdab-9f07d1065c2d\") " pod="openshift-console-operator/console-operator-85b454874b-8xjm5" Sep 08 16:51:25.547948 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.547936 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c66fe27-30fe-4051-9707-acf165ca731f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j5fcl\" (UID: \"0c66fe27-30fe-4051-9707-acf165ca731f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j5fcl" Sep 08 16:51:25.548191 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.547960 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5977d6b9-f03a-40f2-bdab-9f07d1065c2d-trusted-ca\") pod \"console-operator-85b454874b-8xjm5\" (UID: \"5977d6b9-f03a-40f2-bdab-9f07d1065c2d\") " pod="openshift-console-operator/console-operator-85b454874b-8xjm5" Sep 08 16:51:25.548191 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.547985 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec6cae4f-377a-4717-acb8-63166637113a-service-ca-bundle\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:25.548191 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.548010 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-metrics-certs\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:25.548191 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.548039 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1c8b5c6-12f4-451f-93ac-7e1878b355be-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-dld4j\" (UID: \"c1c8b5c6-12f4-451f-93ac-7e1878b355be\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" Sep 08 16:51:25.548191 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.548094 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt7dt\" (UniqueName: \"kubernetes.io/projected/5977d6b9-f03a-40f2-bdab-9f07d1065c2d-kube-api-access-dt7dt\") pod \"console-operator-85b454874b-8xjm5\" (UID: \"5977d6b9-f03a-40f2-bdab-9f07d1065c2d\") " pod="openshift-console-operator/console-operator-85b454874b-8xjm5" Sep 08 16:51:25.548191 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.548121 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0d648a28-280d-4584-a061-704134b80708-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-ngvwg\" (UID: \"0d648a28-280d-4584-a061-704134b80708\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg" Sep 08 16:51:25.548191 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.548159 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1c8b5c6-12f4-451f-93ac-7e1878b355be-tmp\") pod \"insights-operator-76bcd6ccf5-dld4j\" (UID: \"c1c8b5c6-12f4-451f-93ac-7e1878b355be\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" Sep 08 16:51:25.548191 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.548189 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d648a28-280d-4584-a061-704134b80708-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ngvwg\" (UID: \"0d648a28-280d-4584-a061-704134b80708\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg" Sep 08 16:51:25.548599 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.548218 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx4cc\" (UniqueName: \"kubernetes.io/projected/706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5-kube-api-access-dx4cc\") pod \"service-ca-operator-859f94c4b-t48r4\" (UID: \"706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t48r4" Sep 08 16:51:25.548599 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.548245 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-stats-auth\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:25.548599 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.548253 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5-config\") pod \"service-ca-operator-859f94c4b-t48r4\" (UID: \"706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t48r4" Sep 08 16:51:25.548599 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:25.548302 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec6cae4f-377a-4717-acb8-63166637113a-service-ca-bundle podName:ec6cae4f-377a-4717-acb8-63166637113a nodeName:}" failed. No retries permitted until 2026-09-08 16:51:26.048278122 +0000 UTC m=+48.785109230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec6cae4f-377a-4717-acb8-63166637113a-service-ca-bundle") pod "router-default-858549dccb-6vpw7" (UID: "ec6cae4f-377a-4717-acb8-63166637113a") : configmap references non-existent config key: service-ca.crt Sep 08 16:51:25.548599 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:25.548390 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 08 16:51:25.548599 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:25.548477 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c66fe27-30fe-4051-9707-acf165ca731f-samples-operator-tls podName:0c66fe27-30fe-4051-9707-acf165ca731f nodeName:}" failed. No retries permitted until 2026-09-08 16:51:26.048461817 +0000 UTC m=+48.785292929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0c66fe27-30fe-4051-9707-acf165ca731f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-j5fcl" (UID: "0c66fe27-30fe-4051-9707-acf165ca731f") : secret "samples-operator-tls" not found Sep 08 16:51:25.548978 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.548954 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5977d6b9-f03a-40f2-bdab-9f07d1065c2d-config\") pod \"console-operator-85b454874b-8xjm5\" (UID: \"5977d6b9-f03a-40f2-bdab-9f07d1065c2d\") " pod="openshift-console-operator/console-operator-85b454874b-8xjm5" Sep 08 16:51:25.549134 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.549116 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0d648a28-280d-4584-a061-704134b80708-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-ngvwg\" (UID: \"0d648a28-280d-4584-a061-704134b80708\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg" Sep 08 16:51:25.549207 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:25.549198 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 16:51:25.549260 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:25.549243 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d648a28-280d-4584-a061-704134b80708-cluster-monitoring-operator-tls podName:0d648a28-280d-4584-a061-704134b80708 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:26.04922574 +0000 UTC m=+48.786056836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0d648a28-280d-4584-a061-704134b80708-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-ngvwg" (UID: "0d648a28-280d-4584-a061-704134b80708") : secret "cluster-monitoring-operator-tls" not found Sep 08 16:51:25.549502 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.549476 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5977d6b9-f03a-40f2-bdab-9f07d1065c2d-trusted-ca\") pod \"console-operator-85b454874b-8xjm5\" (UID: \"5977d6b9-f03a-40f2-bdab-9f07d1065c2d\") " pod="openshift-console-operator/console-operator-85b454874b-8xjm5" Sep 08 16:51:25.549598 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:25.549512 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 08 16:51:25.549661 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:25.549635 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-metrics-certs podName:ec6cae4f-377a-4717-acb8-63166637113a nodeName:}" failed. No retries permitted until 2026-09-08 16:51:26.049619398 +0000 UTC m=+48.786450497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-metrics-certs") pod "router-default-858549dccb-6vpw7" (UID: "ec6cae4f-377a-4717-acb8-63166637113a") : secret "router-metrics-certs-default" not found Sep 08 16:51:25.550710 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.550690 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5-serving-cert\") pod \"service-ca-operator-859f94c4b-t48r4\" (UID: \"706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t48r4" Sep 08 16:51:25.550848 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.550820 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-default-certificate\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:25.550924 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.550912 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-64p5d" Sep 08 16:51:25.551777 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.551754 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5977d6b9-f03a-40f2-bdab-9f07d1065c2d-serving-cert\") pod \"console-operator-85b454874b-8xjm5\" (UID: \"5977d6b9-f03a-40f2-bdab-9f07d1065c2d\") " pod="openshift-console-operator/console-operator-85b454874b-8xjm5" Sep 08 16:51:25.552089 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.552067 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-stats-auth\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:25.555803 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.555768 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb26j\" (UniqueName: \"kubernetes.io/projected/0d648a28-280d-4584-a061-704134b80708-kube-api-access-pb26j\") pod \"cluster-monitoring-operator-6cdcc96965-ngvwg\" (UID: \"0d648a28-280d-4584-a061-704134b80708\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg" Sep 08 16:51:25.556012 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.555994 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc58g\" (UniqueName: \"kubernetes.io/projected/0c66fe27-30fe-4051-9707-acf165ca731f-kube-api-access-zc58g\") pod \"cluster-samples-operator-7cdcfcf76c-j5fcl\" (UID: \"0c66fe27-30fe-4051-9707-acf165ca731f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j5fcl" Sep 08 16:51:25.556973 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.556949 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx4cc\" (UniqueName: \"kubernetes.io/projected/706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5-kube-api-access-dx4cc\") pod \"service-ca-operator-859f94c4b-t48r4\" (UID: \"706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t48r4" Sep 08 16:51:25.558207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.558189 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt7dt\" (UniqueName: \"kubernetes.io/projected/5977d6b9-f03a-40f2-bdab-9f07d1065c2d-kube-api-access-dt7dt\") pod \"console-operator-85b454874b-8xjm5\" (UID: \"5977d6b9-f03a-40f2-bdab-9f07d1065c2d\") " pod="openshift-console-operator/console-operator-85b454874b-8xjm5" Sep 08 16:51:25.558287 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.558193 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xnv9\" (UniqueName: \"kubernetes.io/projected/ec6cae4f-377a-4717-acb8-63166637113a-kube-api-access-5xnv9\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:25.591186 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.590711 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-8xjm5" Sep 08 16:51:25.649672 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.649634 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c1c8b5c6-12f4-451f-93ac-7e1878b355be-snapshots\") pod \"insights-operator-76bcd6ccf5-dld4j\" (UID: \"c1c8b5c6-12f4-451f-93ac-7e1878b355be\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" Sep 08 16:51:25.649768 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.649683 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1c8b5c6-12f4-451f-93ac-7e1878b355be-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-dld4j\" (UID: \"c1c8b5c6-12f4-451f-93ac-7e1878b355be\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" Sep 08 16:51:25.649768 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.649718 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnztf\" (UniqueName: \"kubernetes.io/projected/c1c8b5c6-12f4-451f-93ac-7e1878b355be-kube-api-access-mnztf\") pod \"insights-operator-76bcd6ccf5-dld4j\" (UID: \"c1c8b5c6-12f4-451f-93ac-7e1878b355be\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" Sep 08 16:51:25.649768 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.649754 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1c8b5c6-12f4-451f-93ac-7e1878b355be-serving-cert\") pod \"insights-operator-76bcd6ccf5-dld4j\" (UID: \"c1c8b5c6-12f4-451f-93ac-7e1878b355be\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" Sep 08 16:51:25.650004 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.649976 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1c8b5c6-12f4-451f-93ac-7e1878b355be-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-dld4j\" (UID: \"c1c8b5c6-12f4-451f-93ac-7e1878b355be\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" Sep 08 16:51:25.650095 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.650073 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1c8b5c6-12f4-451f-93ac-7e1878b355be-tmp\") pod \"insights-operator-76bcd6ccf5-dld4j\" (UID: \"c1c8b5c6-12f4-451f-93ac-7e1878b355be\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" Sep 08 16:51:25.650508 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.650439 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c1c8b5c6-12f4-451f-93ac-7e1878b355be-snapshots\") pod \"insights-operator-76bcd6ccf5-dld4j\" (UID: \"c1c8b5c6-12f4-451f-93ac-7e1878b355be\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" Sep 08 16:51:25.650508 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.650443 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1c8b5c6-12f4-451f-93ac-7e1878b355be-tmp\") pod \"insights-operator-76bcd6ccf5-dld4j\" (UID: \"c1c8b5c6-12f4-451f-93ac-7e1878b355be\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" Sep 08 16:51:25.650737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.650713 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1c8b5c6-12f4-451f-93ac-7e1878b355be-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-dld4j\" (UID: \"c1c8b5c6-12f4-451f-93ac-7e1878b355be\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" Sep 08 16:51:25.650906 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.650821 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t48r4" Sep 08 16:51:25.650998 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.650919 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1c8b5c6-12f4-451f-93ac-7e1878b355be-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-dld4j\" (UID: \"c1c8b5c6-12f4-451f-93ac-7e1878b355be\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" Sep 08 16:51:25.653226 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.652791 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1c8b5c6-12f4-451f-93ac-7e1878b355be-serving-cert\") pod \"insights-operator-76bcd6ccf5-dld4j\" (UID: \"c1c8b5c6-12f4-451f-93ac-7e1878b355be\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" Sep 08 16:51:25.658528 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.658507 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnztf\" (UniqueName: \"kubernetes.io/projected/c1c8b5c6-12f4-451f-93ac-7e1878b355be-kube-api-access-mnztf\") pod \"insights-operator-76bcd6ccf5-dld4j\" (UID: \"c1c8b5c6-12f4-451f-93ac-7e1878b355be\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" Sep 08 16:51:25.686074 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.686034 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" Sep 08 16:51:25.809041 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.809013 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-r87s8"] Sep 08 16:51:25.811476 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:51:25.811452 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e261a0_1dcc_45a8_b32a_48673c0f9482.slice/crio-40a31cf18228ada5c4844aae57389b2774f27237a145907e49399b09de9e1f7b WatchSource:0}: Error finding container 40a31cf18228ada5c4844aae57389b2774f27237a145907e49399b09de9e1f7b: Status 404 returned error can't find the container with id 40a31cf18228ada5c4844aae57389b2774f27237a145907e49399b09de9e1f7b Sep 08 16:51:25.830344 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.830325 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2b5lx"] Sep 08 16:51:25.832909 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:51:25.832887 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode827254b_0e20_4391_8339_3f6541535bc9.slice/crio-98c88415561d60bba96cdb2a72a321a22b5898e924b304090813abfda5978cf7 WatchSource:0}: Error finding container 98c88415561d60bba96cdb2a72a321a22b5898e924b304090813abfda5978cf7: Status 404 returned error can't find the container with id 98c88415561d60bba96cdb2a72a321a22b5898e924b304090813abfda5978cf7 Sep 08 16:51:25.879119 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.879093 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-64p5d"] Sep 08 16:51:25.881319 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:51:25.881297 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2b4e2c_8c9e_459a_bdb5_c0065e7334e4.slice/crio-eb68adca6650d6a52eb75771406b63947baae63c48b7fc88e1d4445a3af75cf7 WatchSource:0}: Error finding container eb68adca6650d6a52eb75771406b63947baae63c48b7fc88e1d4445a3af75cf7: Status 404 returned error can't find the container with id eb68adca6650d6a52eb75771406b63947baae63c48b7fc88e1d4445a3af75cf7 Sep 08 16:51:25.926850 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.926829 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-8xjm5"] Sep 08 16:51:25.929322 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:51:25.929302 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5977d6b9_f03a_40f2_bdab_9f07d1065c2d.slice/crio-687921f47071b21ab7a0db6b2c066c09323d7bfdff7447d000041d361401b73c WatchSource:0}: Error finding container 687921f47071b21ab7a0db6b2c066c09323d7bfdff7447d000041d361401b73c: Status 404 returned error can't find the container with id 687921f47071b21ab7a0db6b2c066c09323d7bfdff7447d000041d361401b73c Sep 08 16:51:25.964899 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.964840 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-r87s8" event={"ID":"38e261a0-1dcc-45a8-b32a-48673c0f9482","Type":"ContainerStarted","Data":"40a31cf18228ada5c4844aae57389b2774f27237a145907e49399b09de9e1f7b"} Sep 08 16:51:25.965970 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.965946 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8xjm5" event={"ID":"5977d6b9-f03a-40f2-bdab-9f07d1065c2d","Type":"ContainerStarted","Data":"687921f47071b21ab7a0db6b2c066c09323d7bfdff7447d000041d361401b73c"} Sep 08 16:51:25.967012 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.966989 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-64p5d" event={"ID":"dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4","Type":"ContainerStarted","Data":"eb68adca6650d6a52eb75771406b63947baae63c48b7fc88e1d4445a3af75cf7"} Sep 08 16:51:25.968007 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.967988 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2b5lx" event={"ID":"e827254b-0e20-4391-8339-3f6541535bc9","Type":"ContainerStarted","Data":"98c88415561d60bba96cdb2a72a321a22b5898e924b304090813abfda5978cf7"} Sep 08 16:51:25.970611 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.970591 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-t48r4"] Sep 08 16:51:25.973095 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:51:25.973076 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706c56f3_ffc9_46a5_8bcd_d7a45ecb1ad5.slice/crio-6f0dc4648aaa531a4c16f8322460d70481c0dad6c6225db2f560c59c369bebed WatchSource:0}: Error finding container 6f0dc4648aaa531a4c16f8322460d70481c0dad6c6225db2f560c59c369bebed: Status 404 returned error can't find the container with id 6f0dc4648aaa531a4c16f8322460d70481c0dad6c6225db2f560c59c369bebed Sep 08 16:51:25.996003 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:25.995977 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-dld4j"] Sep 08 16:51:26.000379 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:51:26.000357 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c8b5c6_12f4_451f_93ac_7e1878b355be.slice/crio-a3c156dcfc93b0ff153f03de41c884bf16c0fa6aab4aef18834c030e2ddba3da WatchSource:0}: Error finding container a3c156dcfc93b0ff153f03de41c884bf16c0fa6aab4aef18834c030e2ddba3da: Status 404 returned error can't find the container with id a3c156dcfc93b0ff153f03de41c884bf16c0fa6aab4aef18834c030e2ddba3da Sep 08 16:51:26.053921 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:26.053903 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c66fe27-30fe-4051-9707-acf165ca731f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j5fcl\" (UID: \"0c66fe27-30fe-4051-9707-acf165ca731f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j5fcl" Sep 08 16:51:26.054032 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:26.053930 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec6cae4f-377a-4717-acb8-63166637113a-service-ca-bundle\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:26.054032 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:26.053956 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-metrics-certs\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:26.054032 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:26.054020 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 08 16:51:26.054180 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:26.054021 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 08 16:51:26.054180 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:26.054073 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec6cae4f-377a-4717-acb8-63166637113a-service-ca-bundle podName:ec6cae4f-377a-4717-acb8-63166637113a nodeName:}" failed. No retries permitted until 2026-09-08 16:51:27.054051181 +0000 UTC m=+49.790882286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec6cae4f-377a-4717-acb8-63166637113a-service-ca-bundle") pod "router-default-858549dccb-6vpw7" (UID: "ec6cae4f-377a-4717-acb8-63166637113a") : configmap references non-existent config key: service-ca.crt Sep 08 16:51:26.054180 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:26.054123 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c66fe27-30fe-4051-9707-acf165ca731f-samples-operator-tls podName:0c66fe27-30fe-4051-9707-acf165ca731f nodeName:}" failed. No retries permitted until 2026-09-08 16:51:27.054103635 +0000 UTC m=+49.790934732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0c66fe27-30fe-4051-9707-acf165ca731f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-j5fcl" (UID: "0c66fe27-30fe-4051-9707-acf165ca731f") : secret "samples-operator-tls" not found Sep 08 16:51:26.054180 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:26.054150 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-metrics-certs podName:ec6cae4f-377a-4717-acb8-63166637113a nodeName:}" failed. No retries permitted until 2026-09-08 16:51:27.054142942 +0000 UTC m=+49.790974036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-metrics-certs") pod "router-default-858549dccb-6vpw7" (UID: "ec6cae4f-377a-4717-acb8-63166637113a") : secret "router-metrics-certs-default" not found Sep 08 16:51:26.054384 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:26.054185 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d648a28-280d-4584-a061-704134b80708-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ngvwg\" (UID: \"0d648a28-280d-4584-a061-704134b80708\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg" Sep 08 16:51:26.054384 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:26.054281 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 16:51:26.054384 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:26.054312 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d648a28-280d-4584-a061-704134b80708-cluster-monitoring-operator-tls podName:0d648a28-280d-4584-a061-704134b80708 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:27.054301275 +0000 UTC m=+49.791132371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0d648a28-280d-4584-a061-704134b80708-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-ngvwg" (UID: "0d648a28-280d-4584-a061-704134b80708") : secret "cluster-monitoring-operator-tls" not found Sep 08 16:51:26.151041 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:26.151013 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:26.916739 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:26.916699 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:26.974498 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:26.974462 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" event={"ID":"c1c8b5c6-12f4-451f-93ac-7e1878b355be","Type":"ContainerStarted","Data":"a3c156dcfc93b0ff153f03de41c884bf16c0fa6aab4aef18834c030e2ddba3da"} Sep 08 16:51:26.978043 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:26.977990 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t48r4" event={"ID":"706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5","Type":"ContainerStarted","Data":"6f0dc4648aaa531a4c16f8322460d70481c0dad6c6225db2f560c59c369bebed"} Sep 08 16:51:27.067078 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:27.066008 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c66fe27-30fe-4051-9707-acf165ca731f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j5fcl\" (UID: \"0c66fe27-30fe-4051-9707-acf165ca731f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j5fcl" Sep 08 16:51:27.067078 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:27.066059 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec6cae4f-377a-4717-acb8-63166637113a-service-ca-bundle\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:27.067078 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:27.066086 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-metrics-certs\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:27.067078 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:27.066158 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d648a28-280d-4584-a061-704134b80708-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ngvwg\" (UID: \"0d648a28-280d-4584-a061-704134b80708\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg" Sep 08 16:51:27.067078 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:27.066317 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 16:51:27.067078 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:27.066384 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d648a28-280d-4584-a061-704134b80708-cluster-monitoring-operator-tls podName:0d648a28-280d-4584-a061-704134b80708 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:29.06636544 +0000 UTC m=+51.803196537 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0d648a28-280d-4584-a061-704134b80708-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-ngvwg" (UID: "0d648a28-280d-4584-a061-704134b80708") : secret "cluster-monitoring-operator-tls" not found Sep 08 16:51:27.067078 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:27.066807 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 08 16:51:27.067078 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:27.066865 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c66fe27-30fe-4051-9707-acf165ca731f-samples-operator-tls podName:0c66fe27-30fe-4051-9707-acf165ca731f nodeName:}" failed. No retries permitted until 2026-09-08 16:51:29.066847841 +0000 UTC m=+51.803678951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0c66fe27-30fe-4051-9707-acf165ca731f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-j5fcl" (UID: "0c66fe27-30fe-4051-9707-acf165ca731f") : secret "samples-operator-tls" not found Sep 08 16:51:27.067078 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:27.066951 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec6cae4f-377a-4717-acb8-63166637113a-service-ca-bundle podName:ec6cae4f-377a-4717-acb8-63166637113a nodeName:}" failed. No retries permitted until 2026-09-08 16:51:29.06693914 +0000 UTC m=+51.803770248 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec6cae4f-377a-4717-acb8-63166637113a-service-ca-bundle") pod "router-default-858549dccb-6vpw7" (UID: "ec6cae4f-377a-4717-acb8-63166637113a") : configmap references non-existent config key: service-ca.crt Sep 08 16:51:27.067078 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:27.067010 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 08 16:51:27.067078 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:27.067041 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-metrics-certs podName:ec6cae4f-377a-4717-acb8-63166637113a nodeName:}" failed. No retries permitted until 2026-09-08 16:51:29.067030378 +0000 UTC m=+51.803861476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-metrics-certs") pod "router-default-858549dccb-6vpw7" (UID: "ec6cae4f-377a-4717-acb8-63166637113a") : secret "router-metrics-certs-default" not found Sep 08 16:51:27.468082 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:27.467853 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:27.533557 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:27.533508 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:28.077096 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:28.076710 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls\") pod \"dns-default-69lb7\" (UID: \"8ed0286d-2b34-44d3-95da-2d99e0772a69\") " pod="openshift-dns/dns-default-69lb7" Sep 08 16:51:28.077096 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:28.076844 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 16:51:28.077096 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:28.076900 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:28.077096 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:28.076918 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls podName:8ed0286d-2b34-44d3-95da-2d99e0772a69 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:44.076897706 +0000 UTC m=+66.813728818 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls") pod "dns-default-69lb7" (UID: "8ed0286d-2b34-44d3-95da-2d99e0772a69") : secret "dns-default-metrics-tls" not found Sep 08 16:51:28.077096 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:28.077000 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 16:51:28.077096 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:28.077014 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8ff9cf7-wh72j: secret "image-registry-tls" not found Sep 08 16:51:28.077096 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:28.077061 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls podName:4ec136d4-4af2-4a3b-9462-c21f278d4b4a nodeName:}" failed. No retries permitted until 2026-09-08 16:51:44.077046493 +0000 UTC m=+66.813877608 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls") pod "image-registry-69c8ff9cf7-wh72j" (UID: "4ec136d4-4af2-4a3b-9462-c21f278d4b4a") : secret "image-registry-tls" not found Sep 08 16:51:29.083548 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:29.083506 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d648a28-280d-4584-a061-704134b80708-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ngvwg\" (UID: \"0d648a28-280d-4584-a061-704134b80708\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg" Sep 08 16:51:29.083970 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:29.083630 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c66fe27-30fe-4051-9707-acf165ca731f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j5fcl\" (UID: \"0c66fe27-30fe-4051-9707-acf165ca731f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j5fcl" Sep 08 16:51:29.083970 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:29.083654 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec6cae4f-377a-4717-acb8-63166637113a-service-ca-bundle\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:29.083970 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:29.083661 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 16:51:29.083970 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:29.083671 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-metrics-certs\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:29.083970 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:29.083723 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d648a28-280d-4584-a061-704134b80708-cluster-monitoring-operator-tls podName:0d648a28-280d-4584-a061-704134b80708 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:33.083705855 +0000 UTC m=+55.820536964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0d648a28-280d-4584-a061-704134b80708-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-ngvwg" (UID: "0d648a28-280d-4584-a061-704134b80708") : secret "cluster-monitoring-operator-tls" not found Sep 08 16:51:29.083970 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:29.083745 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 08 16:51:29.083970 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:29.083794 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-metrics-certs podName:ec6cae4f-377a-4717-acb8-63166637113a nodeName:}" failed. No retries permitted until 2026-09-08 16:51:33.083777129 +0000 UTC m=+55.820608236 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-metrics-certs") pod "router-default-858549dccb-6vpw7" (UID: "ec6cae4f-377a-4717-acb8-63166637113a") : secret "router-metrics-certs-default" not found Sep 08 16:51:29.083970 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:29.083821 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec6cae4f-377a-4717-acb8-63166637113a-service-ca-bundle podName:ec6cae4f-377a-4717-acb8-63166637113a nodeName:}" failed. No retries permitted until 2026-09-08 16:51:33.083801648 +0000 UTC m=+55.820632747 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec6cae4f-377a-4717-acb8-63166637113a-service-ca-bundle") pod "router-default-858549dccb-6vpw7" (UID: "ec6cae4f-377a-4717-acb8-63166637113a") : configmap references non-existent config key: service-ca.crt Sep 08 16:51:29.083970 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:29.083863 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 08 16:51:29.083970 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:29.083914 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c66fe27-30fe-4051-9707-acf165ca731f-samples-operator-tls podName:0c66fe27-30fe-4051-9707-acf165ca731f nodeName:}" failed. No retries permitted until 2026-09-08 16:51:33.083902615 +0000 UTC m=+55.820733747 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0c66fe27-30fe-4051-9707-acf165ca731f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-j5fcl" (UID: "0c66fe27-30fe-4051-9707-acf165ca731f") : secret "samples-operator-tls" not found Sep 08 16:51:31.165687 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:31.165654 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:31.263912 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:31.263869 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:31.385414 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:31.385342 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:31.448029 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:31.448000 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:31.560588 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:31.560558 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:31.662009 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:31.661940 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:31.761771 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:31.761741 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:31.949591 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:31.949558 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:32.000792 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.000767 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/0.log" Sep 08 16:51:32.000971 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.000813 2565 generic.go:356] "Generic (PLEG): container finished" podID="5977d6b9-f03a-40f2-bdab-9f07d1065c2d" containerID="7b27eb79205ac16c8639f75eec03e378df3481d526ae74e35ecb77082313a051" exitCode=255 Sep 08 16:51:32.000971 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.000890 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8xjm5" event={"ID":"5977d6b9-f03a-40f2-bdab-9f07d1065c2d","Type":"ContainerDied","Data":"7b27eb79205ac16c8639f75eec03e378df3481d526ae74e35ecb77082313a051"} Sep 08 16:51:32.001294 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.001153 2565 scope.go:117] "RemoveContainer" containerID="7b27eb79205ac16c8639f75eec03e378df3481d526ae74e35ecb77082313a051" Sep 08 16:51:32.003297 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.002956 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-64p5d" event={"ID":"dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4","Type":"ContainerStarted","Data":"336a2475daa2445191ce2f0d6e963f2168115c71f8beaff500696521ab80e54c"} Sep 08 16:51:32.005008 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.004957 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2b5lx" event={"ID":"e827254b-0e20-4391-8339-3f6541535bc9","Type":"ContainerStarted","Data":"3f9d9df3404073b3c1cca784a9167b1a0e507520f1455d5f09ba2185f7a1b5ca"} Sep 08 16:51:32.006847 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.006812 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-r87s8" event={"ID":"38e261a0-1dcc-45a8-b32a-48673c0f9482","Type":"ContainerStarted","Data":"9f0d7967d8e968d1742703c6fa2cd9ccd7539e27dbb209626f24e446d0166955"} Sep 08 16:51:32.012034 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.009479 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" event={"ID":"c1c8b5c6-12f4-451f-93ac-7e1878b355be","Type":"ContainerStarted","Data":"b75f8026f871af96fe3fd75d7a6f4cf4f8327c9d39ebb603fbaea99852668134"} Sep 08 16:51:32.012034 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.011318 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t48r4" event={"ID":"706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5","Type":"ContainerStarted","Data":"bf1e3d26843a4f64e66c42ed0dd3f601b85b77d6114cefb0152ff2bbd7eed617"} Sep 08 16:51:32.022913 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.022893 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:32.031362 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.031325 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-r87s8" podStartSLOduration=1.6911985139999999 podStartE2EDuration="7.031312392s" totalImagesPullingTime="5.340113878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:32.030834643 +0000 UTC m=+54.767665760" watchObservedRunningTime="2026-09-08 16:51:32.031312392 +0000 UTC m=+54.768143523" Sep 08 16:51:32.052035 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.051994 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-64p5d" podStartSLOduration=1.827901224 podStartE2EDuration="7.051977015s" totalImagesPullingTime="5.224075791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:32.051368272 +0000 UTC m=+54.788199391" watchObservedRunningTime="2026-09-08 16:51:32.051977015 +0000 UTC m=+54.788808134" Sep 08 16:51:32.068294 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.067344 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t48r4" podStartSLOduration=1.941149464 podStartE2EDuration="7.067328698s" totalImagesPullingTime="5.126179234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:32.066583363 +0000 UTC m=+54.803414480" watchObservedRunningTime="2026-09-08 16:51:32.067328698 +0000 UTC m=+54.804159815" Sep 08 16:51:32.110455 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.109747 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2b5lx" podStartSLOduration=1.846992631 podStartE2EDuration="7.109725805s" totalImagesPullingTime="5.262733174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:32.089716992 +0000 UTC m=+54.826548108" watchObservedRunningTime="2026-09-08 16:51:32.109725805 +0000 UTC m=+54.846556923" Sep 08 16:51:32.110455 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.109896 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" podStartSLOduration=2.010507498 podStartE2EDuration="7.109888381s" totalImagesPullingTime="5.099380883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:32.109147398 +0000 UTC m=+54.845978507" watchObservedRunningTime="2026-09-08 16:51:32.109888381 +0000 UTC m=+54.846719499" Sep 08 16:51:32.180622 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.180590 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:32.281095 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.281001 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:32.724395 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.724351 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:32.800194 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.800155 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:32.874235 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.874205 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v57sb"] Sep 08 16:51:32.878201 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.878183 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:32.880845 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.880824 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 08 16:51:32.880965 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.880824 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vxh7j\"" Sep 08 16:51:32.880965 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.880889 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 08 16:51:32.891562 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.891539 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v57sb"] Sep 08 16:51:32.917081 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.917056 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:32.925104 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.925084 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-crio-socket\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:32.925198 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.925125 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:32.925198 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.925182 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9czrj\" (UniqueName: \"kubernetes.io/projected/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-kube-api-access-9czrj\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:32.925268 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.925210 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-data-volume\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:32.925268 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:32.925248 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:33.016462 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.016382 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/1.log" Sep 08 16:51:33.016833 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.016818 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/0.log" Sep 08 16:51:33.016879 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.016853 2565 generic.go:356] "Generic (PLEG): container finished" podID="5977d6b9-f03a-40f2-bdab-9f07d1065c2d" containerID="74c5ae752677d5f24114a3cc687bc577439cad731f968d9f2c7d0ad1c9e8529b" exitCode=255 Sep 08 16:51:33.016971 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.016951 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8xjm5" event={"ID":"5977d6b9-f03a-40f2-bdab-9f07d1065c2d","Type":"ContainerDied","Data":"74c5ae752677d5f24114a3cc687bc577439cad731f968d9f2c7d0ad1c9e8529b"} Sep 08 16:51:33.017019 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.017008 2565 scope.go:117] "RemoveContainer" containerID="7b27eb79205ac16c8639f75eec03e378df3481d526ae74e35ecb77082313a051" Sep 08 16:51:33.017247 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.017229 2565 scope.go:117] "RemoveContainer" containerID="74c5ae752677d5f24114a3cc687bc577439cad731f968d9f2c7d0ad1c9e8529b" Sep 08 16:51:33.017453 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:33.017408 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-8xjm5_openshift-console-operator(5977d6b9-f03a-40f2-bdab-9f07d1065c2d)\"" pod="openshift-console-operator/console-operator-85b454874b-8xjm5" podUID="5977d6b9-f03a-40f2-bdab-9f07d1065c2d" Sep 08 16:51:33.025949 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.025928 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9czrj\" (UniqueName: \"kubernetes.io/projected/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-kube-api-access-9czrj\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:33.026032 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.025959 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-data-volume\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:33.026032 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.025985 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:33.026125 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.026060 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-crio-socket\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:33.026125 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.026092 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:33.026218 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:33.026162 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 16:51:33.026218 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.026171 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-crio-socket\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:33.026218 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:33.026181 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 16:51:33.026361 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:33.026225 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-kube-rbac-proxy-cm podName:18c23d36-8f8e-415c-b6fe-d08ce9316e6e nodeName:}" failed. No retries permitted until 2026-09-08 16:51:33.526206773 +0000 UTC m=+56.263037889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v57sb" (UID: "18c23d36-8f8e-415c-b6fe-d08ce9316e6e") : configmap "kube-rbac-proxy" not found Sep 08 16:51:33.026361 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:33.026253 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-insights-runtime-extractor-tls podName:18c23d36-8f8e-415c-b6fe-d08ce9316e6e nodeName:}" failed. No retries permitted until 2026-09-08 16:51:33.526236469 +0000 UTC m=+56.263067587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v57sb" (UID: "18c23d36-8f8e-415c-b6fe-d08ce9316e6e") : secret "insights-runtime-extractor-tls" not found Sep 08 16:51:33.026361 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.026327 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-data-volume\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:33.036280 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.036253 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9czrj\" (UniqueName: \"kubernetes.io/projected/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-kube-api-access-9czrj\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:33.127819 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.127321 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c66fe27-30fe-4051-9707-acf165ca731f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j5fcl\" (UID: \"0c66fe27-30fe-4051-9707-acf165ca731f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j5fcl" Sep 08 16:51:33.127819 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.127384 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec6cae4f-377a-4717-acb8-63166637113a-service-ca-bundle\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:33.127819 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.127412 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-metrics-certs\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:33.127819 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:33.127524 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 08 16:51:33.127819 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.127560 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d648a28-280d-4584-a061-704134b80708-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ngvwg\" (UID: \"0d648a28-280d-4584-a061-704134b80708\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg" Sep 08 16:51:33.127819 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:33.127581 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-metrics-certs podName:ec6cae4f-377a-4717-acb8-63166637113a nodeName:}" failed. No retries permitted until 2026-09-08 16:51:41.127564265 +0000 UTC m=+63.864395365 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-metrics-certs") pod "router-default-858549dccb-6vpw7" (UID: "ec6cae4f-377a-4717-acb8-63166637113a") : secret "router-metrics-certs-default" not found Sep 08 16:51:33.127819 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:33.127618 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 08 16:51:33.127819 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:33.127684 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c66fe27-30fe-4051-9707-acf165ca731f-samples-operator-tls podName:0c66fe27-30fe-4051-9707-acf165ca731f nodeName:}" failed. No retries permitted until 2026-09-08 16:51:41.127666215 +0000 UTC m=+63.864497323 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0c66fe27-30fe-4051-9707-acf165ca731f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-j5fcl" (UID: "0c66fe27-30fe-4051-9707-acf165ca731f") : secret "samples-operator-tls" not found Sep 08 16:51:33.127819 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:33.127740 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 16:51:33.127819 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:33.127777 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d648a28-280d-4584-a061-704134b80708-cluster-monitoring-operator-tls podName:0d648a28-280d-4584-a061-704134b80708 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:41.127766039 +0000 UTC m=+63.864597143 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0d648a28-280d-4584-a061-704134b80708-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-ngvwg" (UID: "0d648a28-280d-4584-a061-704134b80708") : secret "cluster-monitoring-operator-tls" not found Sep 08 16:51:33.128207 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:33.127853 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec6cae4f-377a-4717-acb8-63166637113a-service-ca-bundle podName:ec6cae4f-377a-4717-acb8-63166637113a nodeName:}" failed. No retries permitted until 2026-09-08 16:51:41.127843555 +0000 UTC m=+63.864674683 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec6cae4f-377a-4717-acb8-63166637113a-service-ca-bundle") pod "router-default-858549dccb-6vpw7" (UID: "ec6cae4f-377a-4717-acb8-63166637113a") : configmap references non-existent config key: service-ca.crt Sep 08 16:51:33.157193 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.157169 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:33.213143 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.213114 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:33.528083 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.528048 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:33.531890 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.531849 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:33.531988 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.531921 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:33.532045 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:33.532015 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 16:51:33.532045 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:33.532032 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 16:51:33.532140 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:33.532088 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-insights-runtime-extractor-tls podName:18c23d36-8f8e-415c-b6fe-d08ce9316e6e nodeName:}" failed. No retries permitted until 2026-09-08 16:51:34.532068033 +0000 UTC m=+57.268899149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v57sb" (UID: "18c23d36-8f8e-415c-b6fe-d08ce9316e6e") : secret "insights-runtime-extractor-tls" not found Sep 08 16:51:33.532140 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:33.532111 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-kube-rbac-proxy-cm podName:18c23d36-8f8e-415c-b6fe-d08ce9316e6e nodeName:}" failed. No retries permitted until 2026-09-08 16:51:34.532098641 +0000 UTC m=+57.268929749 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v57sb" (UID: "18c23d36-8f8e-415c-b6fe-d08ce9316e6e") : configmap "kube-rbac-proxy" not found Sep 08 16:51:33.627734 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:33.627705 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:34.023853 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:34.023825 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/1.log" Sep 08 16:51:34.024172 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:34.024155 2565 scope.go:117] "RemoveContainer" containerID="74c5ae752677d5f24114a3cc687bc577439cad731f968d9f2c7d0ad1c9e8529b" Sep 08 16:51:34.024356 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:34.024309 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-8xjm5_openshift-console-operator(5977d6b9-f03a-40f2-bdab-9f07d1065c2d)\"" pod="openshift-console-operator/console-operator-85b454874b-8xjm5" podUID="5977d6b9-f03a-40f2-bdab-9f07d1065c2d" Sep 08 16:51:34.540549 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:34.540507 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:34.540960 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:34.540596 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:34.540960 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:34.540664 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 16:51:34.540960 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:34.540710 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 16:51:34.540960 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:34.540718 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-insights-runtime-extractor-tls podName:18c23d36-8f8e-415c-b6fe-d08ce9316e6e nodeName:}" failed. No retries permitted until 2026-09-08 16:51:36.540704373 +0000 UTC m=+59.277535470 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v57sb" (UID: "18c23d36-8f8e-415c-b6fe-d08ce9316e6e") : secret "insights-runtime-extractor-tls" not found Sep 08 16:51:34.540960 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:34.540882 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-kube-rbac-proxy-cm podName:18c23d36-8f8e-415c-b6fe-d08ce9316e6e nodeName:}" failed. No retries permitted until 2026-09-08 16:51:36.540867936 +0000 UTC m=+59.277699044 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v57sb" (UID: "18c23d36-8f8e-415c-b6fe-d08ce9316e6e") : configmap "kube-rbac-proxy" not found Sep 08 16:51:34.580944 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:34.580917 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:35.029072 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:35.029041 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2fv8p_1da361fa-6e95-4a71-bac4-e4e5a2c6d5ef/dns-node-resolver/0.log" Sep 08 16:51:35.045358 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:35.045330 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:35.103230 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:35.103193 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:35.591249 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:35.591210 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-8xjm5" Sep 08 16:51:35.591731 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:35.591306 2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-8xjm5" Sep 08 16:51:35.591731 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:35.591716 2565 scope.go:117] "RemoveContainer" containerID="74c5ae752677d5f24114a3cc687bc577439cad731f968d9f2c7d0ad1c9e8529b" Sep 08 16:51:35.591913 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:35.591891 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-8xjm5_openshift-console-operator(5977d6b9-f03a-40f2-bdab-9f07d1065c2d)\"" pod="openshift-console-operator/console-operator-85b454874b-8xjm5" podUID="5977d6b9-f03a-40f2-bdab-9f07d1065c2d" Sep 08 16:51:36.223341 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:36.223310 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:36.425366 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:36.425336 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rr4fp_92dbe1ae-0e8e-4772-9c97-be594f062d76/node-ca/0.log" Sep 08 16:51:36.557662 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:36.557555 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:36.557662 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:36.557616 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:36.557888 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:36.557726 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 16:51:36.557888 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:36.557749 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 16:51:36.557888 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:36.557800 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-kube-rbac-proxy-cm podName:18c23d36-8f8e-415c-b6fe-d08ce9316e6e nodeName:}" failed. No retries permitted until 2026-09-08 16:51:40.557785061 +0000 UTC m=+63.294616161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v57sb" (UID: "18c23d36-8f8e-415c-b6fe-d08ce9316e6e") : configmap "kube-rbac-proxy" not found Sep 08 16:51:36.557888 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:36.557814 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-insights-runtime-extractor-tls podName:18c23d36-8f8e-415c-b6fe-d08ce9316e6e nodeName:}" failed. No retries permitted until 2026-09-08 16:51:40.557807802 +0000 UTC m=+63.294638895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v57sb" (UID: "18c23d36-8f8e-415c-b6fe-d08ce9316e6e") : secret "insights-runtime-extractor-tls" not found Sep 08 16:51:37.025222 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:37.025191 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ms7mp_b46e7a48-1c3e-4499-b640-c3bd623f857d/serve-healthcheck-canary/0.log" Sep 08 16:51:37.626386 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:37.626349 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-64p5d_dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4/kube-storage-version-migrator-operator/0.log" Sep 08 16:51:37.914079 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:37.914011 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9xxnl" Sep 08 16:51:39.564921 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:39.564881 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:40.589664 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:40.589626 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:40.590100 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:40.589688 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:40.590100 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:40.589778 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 16:51:40.590100 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:40.589854 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-insights-runtime-extractor-tls podName:18c23d36-8f8e-415c-b6fe-d08ce9316e6e nodeName:}" failed. No retries permitted until 2026-09-08 16:51:48.589837007 +0000 UTC m=+71.326668101 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v57sb" (UID: "18c23d36-8f8e-415c-b6fe-d08ce9316e6e") : secret "insights-runtime-extractor-tls" not found Sep 08 16:51:40.590100 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:40.589856 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 16:51:40.590100 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:40.589914 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-kube-rbac-proxy-cm podName:18c23d36-8f8e-415c-b6fe-d08ce9316e6e nodeName:}" failed. No retries permitted until 2026-09-08 16:51:48.589894608 +0000 UTC m=+71.326725703 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v57sb" (UID: "18c23d36-8f8e-415c-b6fe-d08ce9316e6e") : configmap "kube-rbac-proxy" not found Sep 08 16:51:41.195076 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:41.195026 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c66fe27-30fe-4051-9707-acf165ca731f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j5fcl\" (UID: \"0c66fe27-30fe-4051-9707-acf165ca731f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j5fcl" Sep 08 16:51:41.195076 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:41.195071 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec6cae4f-377a-4717-acb8-63166637113a-service-ca-bundle\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:41.195076 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:41.195094 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-metrics-certs\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:41.195361 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:41.195161 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d648a28-280d-4584-a061-704134b80708-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ngvwg\" (UID: \"0d648a28-280d-4584-a061-704134b80708\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg" Sep 08 16:51:41.195361 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:41.195202 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 08 16:51:41.195361 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:41.195233 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec6cae4f-377a-4717-acb8-63166637113a-service-ca-bundle podName:ec6cae4f-377a-4717-acb8-63166637113a nodeName:}" failed. No retries permitted until 2026-09-08 16:51:57.195211295 +0000 UTC m=+79.932042405 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec6cae4f-377a-4717-acb8-63166637113a-service-ca-bundle") pod "router-default-858549dccb-6vpw7" (UID: "ec6cae4f-377a-4717-acb8-63166637113a") : configmap references non-existent config key: service-ca.crt Sep 08 16:51:41.195361 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:41.195276 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 16:51:41.195361 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:41.195279 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 08 16:51:41.195361 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:41.195295 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c66fe27-30fe-4051-9707-acf165ca731f-samples-operator-tls podName:0c66fe27-30fe-4051-9707-acf165ca731f nodeName:}" failed. No retries permitted until 2026-09-08 16:51:57.195272964 +0000 UTC m=+79.932104126 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0c66fe27-30fe-4051-9707-acf165ca731f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-j5fcl" (UID: "0c66fe27-30fe-4051-9707-acf165ca731f") : secret "samples-operator-tls" not found Sep 08 16:51:41.195361 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:41.195330 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d648a28-280d-4584-a061-704134b80708-cluster-monitoring-operator-tls podName:0d648a28-280d-4584-a061-704134b80708 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:57.195318101 +0000 UTC m=+79.932149249 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0d648a28-280d-4584-a061-704134b80708-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-ngvwg" (UID: "0d648a28-280d-4584-a061-704134b80708") : secret "cluster-monitoring-operator-tls" not found Sep 08 16:51:41.195361 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:41.195351 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-metrics-certs podName:ec6cae4f-377a-4717-acb8-63166637113a nodeName:}" failed. No retries permitted until 2026-09-08 16:51:57.19534122 +0000 UTC m=+79.932172314 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-metrics-certs") pod "router-default-858549dccb-6vpw7" (UID: "ec6cae4f-377a-4717-acb8-63166637113a") : secret "router-metrics-certs-default" not found Sep 08 16:51:41.338748 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:41.338718 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:41.455576 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:41.455497 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:41.596521 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:41.596489 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:42.112531 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:42.112482 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:42.216342 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:42.216310 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:42.430008 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:42.429981 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:42.607479 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:42.607419 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret\") pod \"global-pull-secret-syncer-jpx5h\" (UID: \"192b1a00-b435-4793-98d5-d51cf7b036d1\") " pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:51:42.607886 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:42.607498 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs\") pod \"network-metrics-daemon-qbb9j\" (UID: \"5d001365-79a1-4fec-bcf8-3abf7309f1ff\") " pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:51:42.609747 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:42.609725 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 16:51:42.609855 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:42.609732 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 16:51:42.618567 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:42.618545 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 08 16:51:42.618634 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:42.618603 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs podName:5d001365-79a1-4fec-bcf8-3abf7309f1ff nodeName:}" failed. No retries permitted until 2026-09-08 16:52:46.618585818 +0000 UTC m=+129.355416917 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs") pod "network-metrics-daemon-qbb9j" (UID: "5d001365-79a1-4fec-bcf8-3abf7309f1ff") : secret "metrics-daemon-secret" not found Sep 08 16:51:42.620695 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:42.620677 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/192b1a00-b435-4793-98d5-d51cf7b036d1-original-pull-secret\") pod \"global-pull-secret-syncer-jpx5h\" (UID: \"192b1a00-b435-4793-98d5-d51cf7b036d1\") " pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:51:42.708600 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:42.708534 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pk2b\" (UniqueName: \"kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b\") pod \"network-check-target-lzqb8\" (UID: \"e59c8379-7774-4209-bda5-cd8c13a1e464\") " pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:51:42.711086 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:42.711066 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pk2b\" (UniqueName: \"kubernetes.io/projected/e59c8379-7774-4209-bda5-cd8c13a1e464-kube-api-access-9pk2b\") pod \"network-check-target-lzqb8\" (UID: \"e59c8379-7774-4209-bda5-cd8c13a1e464\") " pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:51:42.844049 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:42.844015 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:42.875251 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:42.875233 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8582r\"" Sep 08 16:51:42.878931 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:42.878913 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jpx5h" Sep 08 16:51:42.883153 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:42.883137 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:51:43.167227 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:43.167155 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jpx5h"] Sep 08 16:51:43.170365 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:43.170338 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lzqb8"] Sep 08 16:51:43.170499 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:51:43.170391 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192b1a00_b435_4793_98d5_d51cf7b036d1.slice/crio-07939d2c921a277baccf56c754577016e42713db8f07d96543d243476c075c59 WatchSource:0}: Error finding container 07939d2c921a277baccf56c754577016e42713db8f07d96543d243476c075c59: Status 404 returned error can't find the container with id 07939d2c921a277baccf56c754577016e42713db8f07d96543d243476c075c59 Sep 08 16:51:43.628118 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:43.628086 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:44.056205 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:44.056158 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jpx5h" event={"ID":"192b1a00-b435-4793-98d5-d51cf7b036d1","Type":"ContainerStarted","Data":"07939d2c921a277baccf56c754577016e42713db8f07d96543d243476c075c59"} Sep 08 16:51:44.057861 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:44.057834 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lzqb8" event={"ID":"e59c8379-7774-4209-bda5-cd8c13a1e464","Type":"ContainerStarted","Data":"6d00337bc4b069a83ed1326bf9d45a98da66945b4c08e397ba6569097379d801"} Sep 08 16:51:44.057996 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:44.057878 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lzqb8" event={"ID":"e59c8379-7774-4209-bda5-cd8c13a1e464","Type":"ContainerStarted","Data":"7da315b9ccd7c3d402fc05553dd916d6d8571ed91f9c7bf991e35a8e1fe53857"} Sep 08 16:51:44.057996 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:44.057977 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:51:44.073134 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:44.073091 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lzqb8" podStartSLOduration=66.073068634 podStartE2EDuration="1m6.073068634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:44.072343733 +0000 UTC m=+66.809174851" watchObservedRunningTime="2026-09-08 16:51:44.073068634 +0000 UTC m=+66.809899752" Sep 08 16:51:44.076891 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:44.076865 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:44.120239 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:44.120216 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:51:44.120366 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:44.120348 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls\") pod \"dns-default-69lb7\" (UID: \"8ed0286d-2b34-44d3-95da-2d99e0772a69\") " pod="openshift-dns/dns-default-69lb7" Sep 08 16:51:44.120476 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:44.120366 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 16:51:44.120476 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:44.120382 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8ff9cf7-wh72j: secret "image-registry-tls" not found Sep 08 16:51:44.120476 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:44.120447 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls podName:4ec136d4-4af2-4a3b-9462-c21f278d4b4a nodeName:}" failed. No retries permitted until 2026-09-08 16:52:16.120415858 +0000 UTC m=+98.857246957 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls") pod "image-registry-69c8ff9cf7-wh72j" (UID: "4ec136d4-4af2-4a3b-9462-c21f278d4b4a") : secret "image-registry-tls" not found Sep 08 16:51:44.120664 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:44.120513 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 16:51:44.120664 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:44.120571 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls podName:8ed0286d-2b34-44d3-95da-2d99e0772a69 nodeName:}" failed. No retries permitted until 2026-09-08 16:52:16.120555474 +0000 UTC m=+98.857386582 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls") pod "dns-default-69lb7" (UID: "8ed0286d-2b34-44d3-95da-2d99e0772a69") : secret "dns-default-metrics-tls" not found Sep 08 16:51:44.177695 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:44.177657 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:44.277689 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:44.277652 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:44.428174 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:44.428142 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:45.027828 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:45.027797 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:45.427484 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:45.427448 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:46.284838 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:46.284802 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:47.114081 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:47.114055 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:47.217104 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:47.217065 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:48.072153 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:48.072116 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jpx5h" event={"ID":"192b1a00-b435-4793-98d5-d51cf7b036d1","Type":"ContainerStarted","Data":"690d6eb257f9d9dde19bc01d49e134817751102dd6a0c0feb8879f9c879a5697"} Sep 08 16:51:48.088235 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:48.088189 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jpx5h" podStartSLOduration=66.265819674 podStartE2EDuration="1m10.088173778s" totalImagesPullingTime="3.822354104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:48.086483688 +0000 UTC m=+70.823314800" watchObservedRunningTime="2026-09-08 16:51:48.088173778 +0000 UTC m=+70.825004894" Sep 08 16:51:48.245211 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:48.245176 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:48.304134 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:48.304104 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:48.627434 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:48.627387 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:48.660913 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:48.660888 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:48.661019 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:48.660949 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:48.661089 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:48.661036 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 16:51:48.661130 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:48.661103 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-insights-runtime-extractor-tls podName:18c23d36-8f8e-415c-b6fe-d08ce9316e6e nodeName:}" failed. No retries permitted until 2026-09-08 16:52:04.661085812 +0000 UTC m=+87.397916906 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v57sb" (UID: "18c23d36-8f8e-415c-b6fe-d08ce9316e6e") : secret "insights-runtime-extractor-tls" not found Sep 08 16:51:48.661527 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:48.661512 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:51:49.055768 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:49.055727 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:49.624144 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:49.624111 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:49.757865 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:49.757840 2565 scope.go:117] "RemoveContainer" containerID="74c5ae752677d5f24114a3cc687bc577439cad731f968d9f2c7d0ad1c9e8529b" Sep 08 16:51:50.031809 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:50.031770 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:50.080021 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:50.079991 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/2.log" Sep 08 16:51:50.080418 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:50.080400 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/1.log" Sep 08 16:51:50.080540 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:50.080453 2565 generic.go:356] "Generic (PLEG): container finished" podID="5977d6b9-f03a-40f2-bdab-9f07d1065c2d" containerID="7d814343437dd7d0f02bd1dc3c2a3ddf81e87e0a5bbd8a00b0fb22c374a46b2b" exitCode=255 Sep 08 16:51:50.080600 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:50.080524 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8xjm5" event={"ID":"5977d6b9-f03a-40f2-bdab-9f07d1065c2d","Type":"ContainerDied","Data":"7d814343437dd7d0f02bd1dc3c2a3ddf81e87e0a5bbd8a00b0fb22c374a46b2b"} Sep 08 16:51:50.080661 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:50.080644 2565 scope.go:117] "RemoveContainer" containerID="74c5ae752677d5f24114a3cc687bc577439cad731f968d9f2c7d0ad1c9e8529b" Sep 08 16:51:50.080937 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:50.080919 2565 scope.go:117] "RemoveContainer" containerID="7d814343437dd7d0f02bd1dc3c2a3ddf81e87e0a5bbd8a00b0fb22c374a46b2b" Sep 08 16:51:50.081106 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:50.081087 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-8xjm5_openshift-console-operator(5977d6b9-f03a-40f2-bdab-9f07d1065c2d)\"" pod="openshift-console-operator/console-operator-85b454874b-8xjm5" podUID="5977d6b9-f03a-40f2-bdab-9f07d1065c2d" Sep 08 16:51:50.433121 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:50.433091 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:51.072834 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:51.072794 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:51.084848 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:51.084827 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/2.log" Sep 08 16:51:51.127628 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:51.127598 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:51.632142 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:51.632107 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:51.837767 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:51.837735 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:52.029259 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:52.029231 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:55.591870 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:55.591826 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-8xjm5" Sep 08 16:51:55.592255 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:55.591896 2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-8xjm5" Sep 08 16:51:55.592255 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:55.592181 2565 scope.go:117] "RemoveContainer" containerID="7d814343437dd7d0f02bd1dc3c2a3ddf81e87e0a5bbd8a00b0fb22c374a46b2b" Sep 08 16:51:55.592327 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:51:55.592314 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-8xjm5_openshift-console-operator(5977d6b9-f03a-40f2-bdab-9f07d1065c2d)\"" pod="openshift-console-operator/console-operator-85b454874b-8xjm5" podUID="5977d6b9-f03a-40f2-bdab-9f07d1065c2d" Sep 08 16:51:55.928450 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:55.928403 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:55.980220 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:55.980186 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:57.230316 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:57.230272 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c66fe27-30fe-4051-9707-acf165ca731f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j5fcl\" (UID: \"0c66fe27-30fe-4051-9707-acf165ca731f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j5fcl" Sep 08 16:51:57.230316 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:57.230319 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec6cae4f-377a-4717-acb8-63166637113a-service-ca-bundle\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:57.230881 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:57.230340 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-metrics-certs\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:57.230881 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:57.230377 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d648a28-280d-4584-a061-704134b80708-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ngvwg\" (UID: \"0d648a28-280d-4584-a061-704134b80708\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg" Sep 08 16:51:57.231091 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:57.231063 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec6cae4f-377a-4717-acb8-63166637113a-service-ca-bundle\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:57.232819 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:57.232791 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d648a28-280d-4584-a061-704134b80708-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ngvwg\" (UID: \"0d648a28-280d-4584-a061-704134b80708\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg" Sep 08 16:51:57.232819 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:57.232808 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c66fe27-30fe-4051-9707-acf165ca731f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j5fcl\" (UID: \"0c66fe27-30fe-4051-9707-acf165ca731f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j5fcl" Sep 08 16:51:57.232963 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:57.232845 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec6cae4f-377a-4717-acb8-63166637113a-metrics-certs\") pod \"router-default-858549dccb-6vpw7\" (UID: \"ec6cae4f-377a-4717-acb8-63166637113a\") " pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:57.371083 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:57.371049 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cpgpg\"" Sep 08 16:51:57.379660 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:57.379644 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j5fcl" Sep 08 16:51:57.405379 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:57.405357 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-sc6wc\"" Sep 08 16:51:57.414113 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:57.414073 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg" Sep 08 16:51:57.465946 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:57.465909 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-r84x4\"" Sep 08 16:51:57.474717 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:57.474688 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:57.647853 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:57.647826 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j5fcl"] Sep 08 16:51:57.695721 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:57.695697 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg"] Sep 08 16:51:57.698431 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:51:57.698389 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d648a28_280d_4584_a061_704134b80708.slice/crio-f14cf533826363614cdf30a55b8b6e300662f3ce1307a8364f4cf8e164e5ef63 WatchSource:0}: Error finding container f14cf533826363614cdf30a55b8b6e300662f3ce1307a8364f4cf8e164e5ef63: Status 404 returned error can't find the container with id f14cf533826363614cdf30a55b8b6e300662f3ce1307a8364f4cf8e164e5ef63 Sep 08 16:51:57.752807 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:57.752751 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-858549dccb-6vpw7"] Sep 08 16:51:57.756610 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:51:57.756582 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6cae4f_377a_4717_acb8_63166637113a.slice/crio-39890b39d5745ad6a59c174869a87382ce00c25b13977a73336743f903f0c446 WatchSource:0}: Error finding container 39890b39d5745ad6a59c174869a87382ce00c25b13977a73336743f903f0c446: Status 404 returned error can't find the container with id 39890b39d5745ad6a59c174869a87382ce00c25b13977a73336743f903f0c446 Sep 08 16:51:57.909063 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:57.909028 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:58.011979 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:58.011888 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:58.105943 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:58.105908 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-858549dccb-6vpw7" event={"ID":"ec6cae4f-377a-4717-acb8-63166637113a","Type":"ContainerStarted","Data":"30494c17119f9f69f2b6a4695efadb57548e295cad01228504bca13aa315fc16"} Sep 08 16:51:58.106146 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:58.105958 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-858549dccb-6vpw7" event={"ID":"ec6cae4f-377a-4717-acb8-63166637113a","Type":"ContainerStarted","Data":"39890b39d5745ad6a59c174869a87382ce00c25b13977a73336743f903f0c446"} Sep 08 16:51:58.107265 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:58.107243 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg" event={"ID":"0d648a28-280d-4584-a061-704134b80708","Type":"ContainerStarted","Data":"f14cf533826363614cdf30a55b8b6e300662f3ce1307a8364f4cf8e164e5ef63"} Sep 08 16:51:58.107639 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:58.107607 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:58.108475 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:58.108453 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j5fcl" event={"ID":"0c66fe27-30fe-4051-9707-acf165ca731f","Type":"ContainerStarted","Data":"a08a17b8695621c6181d19a956e6d29ab7a5d540c693b479dbf4c3ba93621b63"} Sep 08 16:51:58.125545 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:58.125502 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-858549dccb-6vpw7" podStartSLOduration=33.125487278 podStartE2EDuration="33.125487278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:58.124778808 +0000 UTC m=+80.861609927" watchObservedRunningTime="2026-09-08 16:51:58.125487278 +0000 UTC m=+80.862318391" Sep 08 16:51:58.211926 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:58.211892 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:51:58.475619 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:58.475582 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:58.478504 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:58.478480 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:59.118318 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:59.118279 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:51:59.119732 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:51:59.119710 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-858549dccb-6vpw7" Sep 08 16:52:00.624056 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:00.623967 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:00.723158 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:00.723122 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:00.782298 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:00.782273 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kq45n"] Sep 08 16:52:00.785310 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:00.785293 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kq45n" Sep 08 16:52:00.787788 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:00.787762 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 08 16:52:00.787893 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:00.787772 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6hrlr\"" Sep 08 16:52:00.794542 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:00.794517 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kq45n"] Sep 08 16:52:00.797721 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:00.797700 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:00.864236 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:00.864210 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6d3760ef-42a3-405e-b446-ed0a9c68ffa6-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-kq45n\" (UID: \"6d3760ef-42a3-405e-b446-ed0a9c68ffa6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kq45n" Sep 08 16:52:00.964889 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:00.964865 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6d3760ef-42a3-405e-b446-ed0a9c68ffa6-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-kq45n\" (UID: \"6d3760ef-42a3-405e-b446-ed0a9c68ffa6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kq45n" Sep 08 16:52:00.964990 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:52:00.964977 2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 08 16:52:00.965031 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:52:00.965027 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d3760ef-42a3-405e-b446-ed0a9c68ffa6-tls-certificates podName:6d3760ef-42a3-405e-b446-ed0a9c68ffa6 nodeName:}" failed. No retries permitted until 2026-09-08 16:52:01.465014423 +0000 UTC m=+84.201845516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/6d3760ef-42a3-405e-b446-ed0a9c68ffa6-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-kq45n" (UID: "6d3760ef-42a3-405e-b446-ed0a9c68ffa6") : secret "prometheus-operator-admission-webhook-tls" not found Sep 08 16:52:01.021102 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:01.021067 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:01.127484 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:01.127456 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg" event={"ID":"0d648a28-280d-4584-a061-704134b80708","Type":"ContainerStarted","Data":"32b2cdbda13264cb383b408f96da66471b66b3c420665446950e930a348aff10"} Sep 08 16:52:01.129321 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:01.129296 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j5fcl" event={"ID":"0c66fe27-30fe-4051-9707-acf165ca731f","Type":"ContainerStarted","Data":"199e262616bcfb5442a17d95850bc9955355e04a5f82f540d5b33c7b57a8f54d"} Sep 08 16:52:01.129416 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:01.129338 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j5fcl" event={"ID":"0c66fe27-30fe-4051-9707-acf165ca731f","Type":"ContainerStarted","Data":"f7696c57716cd476d6d331dceafe38a9fbcab7ad4c67f93f3c45c408ef8d9169"} Sep 08 16:52:01.142566 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:01.142524 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ngvwg" podStartSLOduration=33.548042051 podStartE2EDuration="36.142510683s" totalImagesPullingTime="2.594468632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:01.141583754 +0000 UTC m=+83.878414925" watchObservedRunningTime="2026-09-08 16:52:01.142510683 +0000 UTC m=+83.879341798" Sep 08 16:52:01.155793 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:01.155756 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j5fcl" podStartSLOduration=33.547449382 podStartE2EDuration="36.155743108s" totalImagesPullingTime="2.608293726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:01.155342375 +0000 UTC m=+83.892173491" watchObservedRunningTime="2026-09-08 16:52:01.155743108 +0000 UTC m=+83.892574247" Sep 08 16:52:01.469689 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:01.469662 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6d3760ef-42a3-405e-b446-ed0a9c68ffa6-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-kq45n\" (UID: \"6d3760ef-42a3-405e-b446-ed0a9c68ffa6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kq45n" Sep 08 16:52:01.471976 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:01.471951 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6d3760ef-42a3-405e-b446-ed0a9c68ffa6-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-kq45n\" (UID: \"6d3760ef-42a3-405e-b446-ed0a9c68ffa6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kq45n" Sep 08 16:52:01.694135 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:01.694090 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kq45n" Sep 08 16:52:01.962533 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:01.962502 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kq45n"] Sep 08 16:52:01.967373 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:52:01.967347 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3760ef_42a3_405e_b446_ed0a9c68ffa6.slice/crio-a29132ec8e78b201dde4f827d4ff41a596958447792f84038db5fcc647c3d447 WatchSource:0}: Error finding container a29132ec8e78b201dde4f827d4ff41a596958447792f84038db5fcc647c3d447: Status 404 returned error can't find the container with id a29132ec8e78b201dde4f827d4ff41a596958447792f84038db5fcc647c3d447 Sep 08 16:52:02.133962 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:02.133885 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kq45n" event={"ID":"6d3760ef-42a3-405e-b446-ed0a9c68ffa6","Type":"ContainerStarted","Data":"a29132ec8e78b201dde4f827d4ff41a596958447792f84038db5fcc647c3d447"} Sep 08 16:52:03.378548 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:03.378508 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:03.438399 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:03.438318 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:04.142807 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:04.142772 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kq45n" event={"ID":"6d3760ef-42a3-405e-b446-ed0a9c68ffa6","Type":"ContainerStarted","Data":"2377338bab2e2606b24c9f539d489f35c12f9ea09190f7e1571be09955323fd3"} Sep 08 16:52:04.143010 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:04.142974 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kq45n" Sep 08 16:52:04.147943 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:04.147924 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kq45n" Sep 08 16:52:04.159514 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:04.159474 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kq45n" podStartSLOduration=2.950675721 podStartE2EDuration="4.159459293s" totalImagesPullingTime="1.208783572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:04.159442666 +0000 UTC m=+86.896273776" watchObservedRunningTime="2026-09-08 16:52:04.159459293 +0000 UTC m=+86.896290409" Sep 08 16:52:04.420006 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:04.419975 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:04.487790 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:04.487749 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:04.694330 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:04.694241 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:52:04.696509 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:04.696488 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18c23d36-8f8e-415c-b6fe-d08ce9316e6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v57sb\" (UID: \"18c23d36-8f8e-415c-b6fe-d08ce9316e6e\") " pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:52:04.820069 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:04.820034 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:04.921194 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:04.921159 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:04.989638 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:04.989560 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vxh7j\"" Sep 08 16:52:04.997737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:04.997710 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v57sb" Sep 08 16:52:05.282774 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:05.282699 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:05.304908 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:05.304882 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v57sb"] Sep 08 16:52:05.307757 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:52:05.307731 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c23d36_8f8e_415c_b6fe_d08ce9316e6e.slice/crio-0a3e522556e3260ca6b1c2a1c6519aecd3b0ad5db5de48f542d7829dd5c9c7ee WatchSource:0}: Error finding container 0a3e522556e3260ca6b1c2a1c6519aecd3b0ad5db5de48f542d7829dd5c9c7ee: Status 404 returned error can't find the container with id 0a3e522556e3260ca6b1c2a1c6519aecd3b0ad5db5de48f542d7829dd5c9c7ee Sep 08 16:52:06.152721 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:06.152683 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v57sb" event={"ID":"18c23d36-8f8e-415c-b6fe-d08ce9316e6e","Type":"ContainerStarted","Data":"052617bc9908b3ccd2106d2c8b0efb90e8435e8b6afeffc0a4f1f65a612926bb"} Sep 08 16:52:06.153103 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:06.152748 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v57sb" event={"ID":"18c23d36-8f8e-415c-b6fe-d08ce9316e6e","Type":"ContainerStarted","Data":"0a3e522556e3260ca6b1c2a1c6519aecd3b0ad5db5de48f542d7829dd5c9c7ee"} Sep 08 16:52:06.754492 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:06.754468 2565 scope.go:117] "RemoveContainer" containerID="7d814343437dd7d0f02bd1dc3c2a3ddf81e87e0a5bbd8a00b0fb22c374a46b2b" Sep 08 16:52:06.754637 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:52:06.754619 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-8xjm5_openshift-console-operator(5977d6b9-f03a-40f2-bdab-9f07d1065c2d)\"" pod="openshift-console-operator/console-operator-85b454874b-8xjm5" podUID="5977d6b9-f03a-40f2-bdab-9f07d1065c2d" Sep 08 16:52:07.157233 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:07.157195 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v57sb" event={"ID":"18c23d36-8f8e-415c-b6fe-d08ce9316e6e","Type":"ContainerStarted","Data":"dee725d7eaeaf8bc388735b2c9e47b5b2925ba03b847de9ab2ebb961a8f48d50"} Sep 08 16:52:08.162979 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:08.162941 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v57sb" event={"ID":"18c23d36-8f8e-415c-b6fe-d08ce9316e6e","Type":"ContainerStarted","Data":"45bf0974ea2f5290b2491436a5d701dab0d328b431acbda04225ae697c079f75"} Sep 08 16:52:08.182071 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:08.182017 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-v57sb" podStartSLOduration=33.671578321 podStartE2EDuration="36.181998816s" totalImagesPullingTime="2.510420495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:51:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:08.180980453 +0000 UTC m=+90.917811581" watchObservedRunningTime="2026-09-08 16:52:08.181998816 +0000 UTC m=+90.918829932" Sep 08 16:52:08.641094 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:08.641057 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:08.714213 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:08.714181 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:08.786965 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:08.786926 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:08.865513 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:08.865463 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:09.427972 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:09.427935 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:09.703881 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:09.703803 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:09.805412 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:09.805370 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:09.983188 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:09.983105 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:10.084372 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:10.084334 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:10.153522 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:10.153488 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:11.004259 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.004222 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:11.229632 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.229596 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kp8dq"] Sep 08 16:52:11.233157 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.233137 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.235406 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.235382 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 08 16:52:11.235406 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.235403 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 08 16:52:11.235602 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.235392 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 08 16:52:11.235602 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.235444 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-crsks\"" Sep 08 16:52:11.235602 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.235442 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 08 16:52:11.345142 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.345064 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dspsw\" (UniqueName: \"kubernetes.io/projected/dde9ca85-937b-4638-89ba-60ac40e4a5d9-kube-api-access-dspsw\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.345142 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.345097 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dde9ca85-937b-4638-89ba-60ac40e4a5d9-sys\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.345320 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.345143 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dde9ca85-937b-4638-89ba-60ac40e4a5d9-node-exporter-wtmp\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.345320 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.345165 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dde9ca85-937b-4638-89ba-60ac40e4a5d9-node-exporter-tls\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.345320 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.345185 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dde9ca85-937b-4638-89ba-60ac40e4a5d9-node-exporter-textfile\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.345320 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.345236 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dde9ca85-937b-4638-89ba-60ac40e4a5d9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.345320 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.345253 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dde9ca85-937b-4638-89ba-60ac40e4a5d9-root\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.345320 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.345284 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dde9ca85-937b-4638-89ba-60ac40e4a5d9-node-exporter-accelerators-collector-config\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.345320 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.345310 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dde9ca85-937b-4638-89ba-60ac40e4a5d9-metrics-client-ca\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.446580 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.446548 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dde9ca85-937b-4638-89ba-60ac40e4a5d9-node-exporter-wtmp\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.446580 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.446580 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dde9ca85-937b-4638-89ba-60ac40e4a5d9-node-exporter-tls\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.446788 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.446602 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dde9ca85-937b-4638-89ba-60ac40e4a5d9-node-exporter-textfile\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.446788 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.446739 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dde9ca85-937b-4638-89ba-60ac40e4a5d9-node-exporter-wtmp\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.446788 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.446740 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dde9ca85-937b-4638-89ba-60ac40e4a5d9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.446897 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.446805 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dde9ca85-937b-4638-89ba-60ac40e4a5d9-root\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.446897 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.446851 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dde9ca85-937b-4638-89ba-60ac40e4a5d9-node-exporter-accelerators-collector-config\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.446998 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.446908 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dde9ca85-937b-4638-89ba-60ac40e4a5d9-metrics-client-ca\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.446998 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.446922 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dde9ca85-937b-4638-89ba-60ac40e4a5d9-root\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.447098 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.446995 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dspsw\" (UniqueName: \"kubernetes.io/projected/dde9ca85-937b-4638-89ba-60ac40e4a5d9-kube-api-access-dspsw\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.447098 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.447027 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dde9ca85-937b-4638-89ba-60ac40e4a5d9-sys\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.447098 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.447039 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dde9ca85-937b-4638-89ba-60ac40e4a5d9-node-exporter-textfile\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.447222 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.447137 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dde9ca85-937b-4638-89ba-60ac40e4a5d9-sys\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.447515 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.447494 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dde9ca85-937b-4638-89ba-60ac40e4a5d9-metrics-client-ca\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.447574 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.447494 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dde9ca85-937b-4638-89ba-60ac40e4a5d9-node-exporter-accelerators-collector-config\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.448875 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.448854 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dde9ca85-937b-4638-89ba-60ac40e4a5d9-node-exporter-tls\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.448950 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.448941 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dde9ca85-937b-4638-89ba-60ac40e4a5d9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.454761 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.454740 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dspsw\" (UniqueName: \"kubernetes.io/projected/dde9ca85-937b-4638-89ba-60ac40e4a5d9-kube-api-access-dspsw\") pod \"node-exporter-kp8dq\" (UID: \"dde9ca85-937b-4638-89ba-60ac40e4a5d9\") " pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.542004 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.541978 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kp8dq" Sep 08 16:52:11.553478 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:52:11.553449 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde9ca85_937b_4638_89ba_60ac40e4a5d9.slice/crio-dbcfe22af3a7195e220e480db0351d2d4b9704e0dc6eeddb5a76e163b8034ccb WatchSource:0}: Error finding container dbcfe22af3a7195e220e480db0351d2d4b9704e0dc6eeddb5a76e163b8034ccb: Status 404 returned error can't find the container with id dbcfe22af3a7195e220e480db0351d2d4b9704e0dc6eeddb5a76e163b8034ccb Sep 08 16:52:11.718939 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.718906 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:11.784784 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:11.784750 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:12.179783 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:12.179746 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kp8dq" event={"ID":"dde9ca85-937b-4638-89ba-60ac40e4a5d9","Type":"ContainerStarted","Data":"dbcfe22af3a7195e220e480db0351d2d4b9704e0dc6eeddb5a76e163b8034ccb"} Sep 08 16:52:12.207408 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:12.207373 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:12.293062 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:12.293023 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:12.436048 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:12.435965 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:13.184605 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:13.184574 2565 generic.go:356] "Generic (PLEG): container finished" podID="dde9ca85-937b-4638-89ba-60ac40e4a5d9" containerID="0bcf389e1f90fa584f91e47e34ac92f7ff674da2649485b5a5ab0e5b90f0ac82" exitCode=0 Sep 08 16:52:13.185028 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:13.184609 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kp8dq" event={"ID":"dde9ca85-937b-4638-89ba-60ac40e4a5d9","Type":"ContainerDied","Data":"0bcf389e1f90fa584f91e47e34ac92f7ff674da2649485b5a5ab0e5b90f0ac82"} Sep 08 16:52:14.180019 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:14.179979 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:14.190343 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:14.190311 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kp8dq" event={"ID":"dde9ca85-937b-4638-89ba-60ac40e4a5d9","Type":"ContainerStarted","Data":"33a81da220249ffa5a65fce9c5fcb533590edf36ccb5db6b383dad1e7cd8d1d4"} Sep 08 16:52:14.190705 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:14.190372 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kp8dq" event={"ID":"dde9ca85-937b-4638-89ba-60ac40e4a5d9","Type":"ContainerStarted","Data":"ecf9c2a3bc95101193e2eadc52e7d9c8160a54bd27fef02ab0ed40813ba98ec7"} Sep 08 16:52:14.211175 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:14.211123 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kp8dq" podStartSLOduration=1.996783545 podStartE2EDuration="3.211107081s" totalImagesPullingTime="1.152889584s" totalInitContainerRuntime="61.433952ms" isStatefulPod=true podCreationTimestamp="2026-09-08 16:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:14.209286855 +0000 UTC m=+96.946117970" watchObservedRunningTime="2026-09-08 16:52:14.211107081 +0000 UTC m=+96.947938196" Sep 08 16:52:14.280635 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:14.280602 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:14.435078 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:14.434981 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:14.574495 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:14.574455 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:15.063502 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:15.063473 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lzqb8" Sep 08 16:52:15.982505 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:15.982390 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:16.188758 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:16.188721 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls\") pod \"dns-default-69lb7\" (UID: \"8ed0286d-2b34-44d3-95da-2d99e0772a69\") " pod="openshift-dns/dns-default-69lb7" Sep 08 16:52:16.188917 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:16.188772 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:52:16.191093 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:16.191071 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ed0286d-2b34-44d3-95da-2d99e0772a69-metrics-tls\") pod \"dns-default-69lb7\" (UID: \"8ed0286d-2b34-44d3-95da-2d99e0772a69\") " pod="openshift-dns/dns-default-69lb7" Sep 08 16:52:16.191199 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:16.191184 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls\") pod \"image-registry-69c8ff9cf7-wh72j\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:52:16.414038 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:16.413960 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-f9rmv\"" Sep 08 16:52:16.422745 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:16.422724 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:52:16.437823 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:16.437800 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zpm9l\"" Sep 08 16:52:16.446372 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:16.446350 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-69lb7" Sep 08 16:52:16.557385 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:16.557355 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:16.703920 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:16.703896 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69c8ff9cf7-wh72j"] Sep 08 16:52:16.706363 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:52:16.706335 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec136d4_4af2_4a3b_9462_c21f278d4b4a.slice/crio-08e719bfa0eed97e4a7237c6028303d0f4252e408fa88de7ef512e0b1db051f5 WatchSource:0}: Error finding container 08e719bfa0eed97e4a7237c6028303d0f4252e408fa88de7ef512e0b1db051f5: Status 404 returned error can't find the container with id 08e719bfa0eed97e4a7237c6028303d0f4252e408fa88de7ef512e0b1db051f5 Sep 08 16:52:16.734478 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:16.734446 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-69lb7"] Sep 08 16:52:16.739443 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:52:16.739398 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed0286d_2b34_44d3_95da_2d99e0772a69.slice/crio-2cfef1655279ddce8bf9aadb63022b6e002ee3e4fe71b9ae02c50e4808b41b3f WatchSource:0}: Error finding container 2cfef1655279ddce8bf9aadb63022b6e002ee3e4fe71b9ae02c50e4808b41b3f: Status 404 returned error can't find the container with id 2cfef1655279ddce8bf9aadb63022b6e002ee3e4fe71b9ae02c50e4808b41b3f Sep 08 16:52:17.202217 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:17.202183 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" event={"ID":"4ec136d4-4af2-4a3b-9462-c21f278d4b4a","Type":"ContainerStarted","Data":"46e5fe18cb242dff61b71e7632ea708f4ee63b7f3b769f58d939c21fa13b9db8"} Sep 08 16:52:17.202664 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:17.202236 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" event={"ID":"4ec136d4-4af2-4a3b-9462-c21f278d4b4a","Type":"ContainerStarted","Data":"08e719bfa0eed97e4a7237c6028303d0f4252e408fa88de7ef512e0b1db051f5"} Sep 08 16:52:17.202664 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:17.202262 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:52:17.205978 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:17.205954 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-69lb7" event={"ID":"8ed0286d-2b34-44d3-95da-2d99e0772a69","Type":"ContainerStarted","Data":"2cfef1655279ddce8bf9aadb63022b6e002ee3e4fe71b9ae02c50e4808b41b3f"} Sep 08 16:52:17.235048 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:17.234998 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" podStartSLOduration=99.23497635 podStartE2EDuration="1m39.23497635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:17.234813183 +0000 UTC m=+99.971644338" watchObservedRunningTime="2026-09-08 16:52:17.23497635 +0000 UTC m=+99.971807468" Sep 08 16:52:17.547621 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:17.547532 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:17.927361 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:17.927326 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:18.311121 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:18.310972 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:18.397860 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:18.397818 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:19.214897 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:19.214800 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-69lb7" event={"ID":"8ed0286d-2b34-44d3-95da-2d99e0772a69","Type":"ContainerStarted","Data":"627963c5b8c03761606733e190f14837836a0c47bde2a304df7ceb9f0e669802"} Sep 08 16:52:19.214897 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:19.214853 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-69lb7" event={"ID":"8ed0286d-2b34-44d3-95da-2d99e0772a69","Type":"ContainerStarted","Data":"6437abff3340bc03c7243577c71c8602530d0a378991e8ecef7d1bd8ca0fb4ec"} Sep 08 16:52:19.215111 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:19.214925 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-69lb7" Sep 08 16:52:19.236000 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:19.235946 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-69lb7" podStartSLOduration=65.286589646 podStartE2EDuration="1m7.235929273s" totalImagesPullingTime="1.949339627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:19.234675776 +0000 UTC m=+101.971506891" watchObservedRunningTime="2026-09-08 16:52:19.235929273 +0000 UTC m=+101.972760388" Sep 08 16:52:19.354671 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:19.354635 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:19.470686 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:19.470602 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:20.525881 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:20.525840 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69c8ff9cf7-wh72j"] Sep 08 16:52:20.698924 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:20.698885 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:20.876707 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:20.876626 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:21.016878 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:21.016840 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:21.754389 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:21.754360 2565 scope.go:117] "RemoveContainer" containerID="7d814343437dd7d0f02bd1dc3c2a3ddf81e87e0a5bbd8a00b0fb22c374a46b2b" Sep 08 16:52:22.227841 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:22.227818 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/2.log" Sep 08 16:52:22.227995 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:22.227899 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8xjm5" event={"ID":"5977d6b9-f03a-40f2-bdab-9f07d1065c2d","Type":"ContainerStarted","Data":"616645db04d4e230c1722569d4c7890ffd3e6d593e9d802cba6ddecceebaa137"} Sep 08 16:52:22.228203 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:22.228177 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-8xjm5" Sep 08 16:52:22.232937 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:22.232918 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-8xjm5" Sep 08 16:52:22.255690 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:22.255649 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-8xjm5" podStartSLOduration=52.082652446 podStartE2EDuration="57.255635729s" totalImagesPullingTime="5.172983283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:22.255275539 +0000 UTC m=+104.992106659" watchObservedRunningTime="2026-09-08 16:52:22.255635729 +0000 UTC m=+104.992466892" Sep 08 16:52:22.500153 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:22.500068 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-rkvsz"] Sep 08 16:52:22.503604 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:22.503583 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-rkvsz" Sep 08 16:52:22.508751 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:22.508728 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-f2mqn\"" Sep 08 16:52:22.508869 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:22.508728 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 08 16:52:22.509513 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:22.509494 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 08 16:52:22.522472 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:22.522443 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-rkvsz"] Sep 08 16:52:22.541492 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:22.541464 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7xmp\" (UniqueName: \"kubernetes.io/projected/b8eda268-da49-429e-bba5-0d983fcb4a74-kube-api-access-t7xmp\") pod \"downloads-85d98c998f-rkvsz\" (UID: \"b8eda268-da49-429e-bba5-0d983fcb4a74\") " pod="openshift-console/downloads-85d98c998f-rkvsz" Sep 08 16:52:22.564594 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:22.564550 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:22.642776 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:22.642756 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7xmp\" (UniqueName: \"kubernetes.io/projected/b8eda268-da49-429e-bba5-0d983fcb4a74-kube-api-access-t7xmp\") pod \"downloads-85d98c998f-rkvsz\" (UID: \"b8eda268-da49-429e-bba5-0d983fcb4a74\") " pod="openshift-console/downloads-85d98c998f-rkvsz" Sep 08 16:52:22.652702 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:22.652678 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7xmp\" (UniqueName: \"kubernetes.io/projected/b8eda268-da49-429e-bba5-0d983fcb4a74-kube-api-access-t7xmp\") pod \"downloads-85d98c998f-rkvsz\" (UID: \"b8eda268-da49-429e-bba5-0d983fcb4a74\") " pod="openshift-console/downloads-85d98c998f-rkvsz" Sep 08 16:52:22.665026 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:22.665001 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:22.761688 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:22.761613 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:22.812737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:22.812713 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-rkvsz" Sep 08 16:52:22.935297 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:22.935267 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:23.040960 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:23.040892 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:23.092789 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:23.092762 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-rkvsz"] Sep 08 16:52:23.095767 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:52:23.095743 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8eda268_da49_429e_bba5_0d983fcb4a74.slice/crio-b5e788ba57997910e465b036a5867243fa3a6d93da015bb04d371367e2d69eaa WatchSource:0}: Error finding container b5e788ba57997910e465b036a5867243fa3a6d93da015bb04d371367e2d69eaa: Status 404 returned error can't find the container with id b5e788ba57997910e465b036a5867243fa3a6d93da015bb04d371367e2d69eaa Sep 08 16:52:23.234244 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:23.234216 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-rkvsz" event={"ID":"b8eda268-da49-429e-bba5-0d983fcb4a74","Type":"ContainerStarted","Data":"b5e788ba57997910e465b036a5867243fa3a6d93da015bb04d371367e2d69eaa"} Sep 08 16:52:23.583735 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:23.583698 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:23.741530 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:23.741487 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:23.887898 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:23.887819 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:24.141409 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:24.141314 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:24.342414 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:24.342380 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:24.960156 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:24.960114 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:25.069739 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:25.069709 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:25.629814 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:25.629775 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:25.937991 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:25.937957 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:26.005895 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:26.005857 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:29.221086 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:29.221052 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-69lb7" Sep 08 16:52:29.240012 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:29.239969 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:29.412786 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:29.412748 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:29.513773 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:29.513702 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:31.957022 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:31.956966 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:32.066367 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:32.066337 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:32.936493 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:32.936461 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:33.041794 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:33.041642 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:34.377640 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.377608 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c99c5f9b6-24b2b"] Sep 08 16:52:34.381710 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.381694 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:34.388755 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.388731 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 08 16:52:34.388876 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.388765 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cm596\"" Sep 08 16:52:34.388876 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.388771 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 08 16:52:34.390290 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.390270 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 08 16:52:34.390376 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.390301 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 08 16:52:34.410602 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.410569 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 08 16:52:34.412438 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.412394 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c99c5f9b6-24b2b"] Sep 08 16:52:34.445163 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.445130 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f4a17c9-7236-4a88-9e26-c99bf5a58569-console-serving-cert\") pod \"console-7c99c5f9b6-24b2b\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:34.445328 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.445171 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f4a17c9-7236-4a88-9e26-c99bf5a58569-oauth-serving-cert\") pod \"console-7c99c5f9b6-24b2b\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:34.445328 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.445198 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b729s\" (UniqueName: \"kubernetes.io/projected/9f4a17c9-7236-4a88-9e26-c99bf5a58569-kube-api-access-b729s\") pod \"console-7c99c5f9b6-24b2b\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:34.445328 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.445296 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f4a17c9-7236-4a88-9e26-c99bf5a58569-service-ca\") pod \"console-7c99c5f9b6-24b2b\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:34.445506 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.445344 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f4a17c9-7236-4a88-9e26-c99bf5a58569-console-oauth-config\") pod \"console-7c99c5f9b6-24b2b\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:34.445506 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.445451 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f4a17c9-7236-4a88-9e26-c99bf5a58569-console-config\") pod \"console-7c99c5f9b6-24b2b\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:34.546512 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.546466 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f4a17c9-7236-4a88-9e26-c99bf5a58569-oauth-serving-cert\") pod \"console-7c99c5f9b6-24b2b\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:34.546512 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.546516 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b729s\" (UniqueName: \"kubernetes.io/projected/9f4a17c9-7236-4a88-9e26-c99bf5a58569-kube-api-access-b729s\") pod \"console-7c99c5f9b6-24b2b\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:34.546751 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.546568 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f4a17c9-7236-4a88-9e26-c99bf5a58569-service-ca\") pod \"console-7c99c5f9b6-24b2b\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:34.546751 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.546638 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f4a17c9-7236-4a88-9e26-c99bf5a58569-console-oauth-config\") pod \"console-7c99c5f9b6-24b2b\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:34.546751 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.546676 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f4a17c9-7236-4a88-9e26-c99bf5a58569-console-config\") pod \"console-7c99c5f9b6-24b2b\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:34.546751 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.546711 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f4a17c9-7236-4a88-9e26-c99bf5a58569-console-serving-cert\") pod \"console-7c99c5f9b6-24b2b\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:34.547249 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.547209 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f4a17c9-7236-4a88-9e26-c99bf5a58569-oauth-serving-cert\") pod \"console-7c99c5f9b6-24b2b\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:34.547346 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.547294 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f4a17c9-7236-4a88-9e26-c99bf5a58569-service-ca\") pod \"console-7c99c5f9b6-24b2b\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:34.547346 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.547324 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f4a17c9-7236-4a88-9e26-c99bf5a58569-console-config\") pod \"console-7c99c5f9b6-24b2b\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:34.548978 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.548947 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f4a17c9-7236-4a88-9e26-c99bf5a58569-console-oauth-config\") pod \"console-7c99c5f9b6-24b2b\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:34.549088 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.549067 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f4a17c9-7236-4a88-9e26-c99bf5a58569-console-serving-cert\") pod \"console-7c99c5f9b6-24b2b\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:34.558678 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.558657 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b729s\" (UniqueName: \"kubernetes.io/projected/9f4a17c9-7236-4a88-9e26-c99bf5a58569-kube-api-access-b729s\") pod \"console-7c99c5f9b6-24b2b\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:34.691833 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.691794 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:34.969837 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.969746 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:34.990990 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:34.990906 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c99c5f9b6-24b2b"] Sep 08 16:52:35.060955 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:35.060915 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:39.362329 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:39.362277 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:39.460222 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:39.460180 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:40.532884 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:40.532855 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:52:41.035898 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:41.035845 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:41.130027 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:41.129990 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:41.745904 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:52:41.745865 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4a17c9_7236_4a88_9e26_c99bf5a58569.slice/crio-ae1c219c3a7c672a225002f3f39f66a09dbb08ec963890803da7d16148456fd2 WatchSource:0}: Error finding container ae1c219c3a7c672a225002f3f39f66a09dbb08ec963890803da7d16148456fd2: Status 404 returned error can't find the container with id ae1c219c3a7c672a225002f3f39f66a09dbb08ec963890803da7d16148456fd2 Sep 08 16:52:42.316914 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:42.316875 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c99c5f9b6-24b2b" event={"ID":"9f4a17c9-7236-4a88-9e26-c99bf5a58569","Type":"ContainerStarted","Data":"ae1c219c3a7c672a225002f3f39f66a09dbb08ec963890803da7d16148456fd2"} Sep 08 16:52:42.318879 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:42.318841 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-rkvsz" event={"ID":"b8eda268-da49-429e-bba5-0d983fcb4a74","Type":"ContainerStarted","Data":"461349dc5df0b9824afb6ab798760a75ee333b141731aac2fef19f8a983ff7da"} Sep 08 16:52:42.319021 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:42.318905 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-rkvsz" Sep 08 16:52:42.320808 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:42.320775 2565 generic.go:356] "Generic (PLEG): container finished" podID="dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4" containerID="336a2475daa2445191ce2f0d6e963f2168115c71f8beaff500696521ab80e54c" exitCode=0 Sep 08 16:52:42.320946 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:42.320844 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-64p5d" event={"ID":"dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4","Type":"ContainerDied","Data":"336a2475daa2445191ce2f0d6e963f2168115c71f8beaff500696521ab80e54c"} Sep 08 16:52:42.321182 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:42.321162 2565 scope.go:117] "RemoveContainer" containerID="336a2475daa2445191ce2f0d6e963f2168115c71f8beaff500696521ab80e54c" Sep 08 16:52:42.331386 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:42.331364 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-rkvsz" Sep 08 16:52:42.339718 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:42.339638 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-rkvsz" podStartSLOduration=1.5861228189999999 podStartE2EDuration="20.339618792s" totalImagesPullingTime="18.753495973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:52:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:42.339382736 +0000 UTC m=+125.076213884" watchObservedRunningTime="2026-09-08 16:52:42.339618792 +0000 UTC m=+125.076449910" Sep 08 16:52:43.330053 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:43.329407 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-64p5d" event={"ID":"dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4","Type":"ContainerStarted","Data":"9292578c187e816caf65b480fa459b0dfb2bfae9ff5f736b6bdc0e54449b38d3"} Sep 08 16:52:43.709197 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:43.709095 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:43.811154 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:43.811082 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:44.581570 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:44.581483 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:44.659846 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:44.659814 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:45.352343 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.352301 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7785b5978d-qd69v"] Sep 08 16:52:45.364487 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.364459 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.371890 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.371857 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7785b5978d-qd69v"] Sep 08 16:52:45.372592 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.372566 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 08 16:52:45.459577 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.459546 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-trusted-ca-bundle\") pod \"console-7785b5978d-qd69v\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.459748 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.459616 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4vq7\" (UniqueName: \"kubernetes.io/projected/7df39976-8653-457c-9eae-d4d09ea18ad5-kube-api-access-j4vq7\") pod \"console-7785b5978d-qd69v\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.459748 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.459650 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7df39976-8653-457c-9eae-d4d09ea18ad5-console-oauth-config\") pod \"console-7785b5978d-qd69v\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.459748 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.459731 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-oauth-serving-cert\") pod \"console-7785b5978d-qd69v\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.459902 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.459780 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-console-config\") pod \"console-7785b5978d-qd69v\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.459902 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.459812 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-service-ca\") pod \"console-7785b5978d-qd69v\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.459902 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.459883 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7df39976-8653-457c-9eae-d4d09ea18ad5-console-serving-cert\") pod \"console-7785b5978d-qd69v\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.545014 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.544971 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" podUID="4ec136d4-4af2-4a3b-9462-c21f278d4b4a" containerName="registry" containerID="cri-o://46e5fe18cb242dff61b71e7632ea708f4ee63b7f3b769f58d939c21fa13b9db8" gracePeriod=30 Sep 08 16:52:45.560778 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.560748 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-trusted-ca-bundle\") pod \"console-7785b5978d-qd69v\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.560904 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.560817 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4vq7\" (UniqueName: \"kubernetes.io/projected/7df39976-8653-457c-9eae-d4d09ea18ad5-kube-api-access-j4vq7\") pod \"console-7785b5978d-qd69v\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.560904 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.560856 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7df39976-8653-457c-9eae-d4d09ea18ad5-console-oauth-config\") pod \"console-7785b5978d-qd69v\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.560904 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.560894 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-oauth-serving-cert\") pod \"console-7785b5978d-qd69v\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.561071 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.560932 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-console-config\") pod \"console-7785b5978d-qd69v\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.561071 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.560960 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-service-ca\") pod \"console-7785b5978d-qd69v\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.561071 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.560985 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7df39976-8653-457c-9eae-d4d09ea18ad5-console-serving-cert\") pod \"console-7785b5978d-qd69v\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.561728 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.561677 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-oauth-serving-cert\") pod \"console-7785b5978d-qd69v\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.561728 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.561691 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-console-config\") pod \"console-7785b5978d-qd69v\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.562616 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.561905 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-service-ca\") pod \"console-7785b5978d-qd69v\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.562698 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.562621 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-trusted-ca-bundle\") pod \"console-7785b5978d-qd69v\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.563908 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.563744 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7df39976-8653-457c-9eae-d4d09ea18ad5-console-serving-cert\") pod \"console-7785b5978d-qd69v\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.563908 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.563826 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7df39976-8653-457c-9eae-d4d09ea18ad5-console-oauth-config\") pod \"console-7785b5978d-qd69v\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.569961 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.569938 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4vq7\" (UniqueName: \"kubernetes.io/projected/7df39976-8653-457c-9eae-d4d09ea18ad5-kube-api-access-j4vq7\") pod \"console-7785b5978d-qd69v\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.676316 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.676276 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:45.958918 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:45.958887 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:52:46.058202 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.058001 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7785b5978d-qd69v"] Sep 08 16:52:46.061129 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:52:46.061100 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df39976_8653_457c_9eae_d4d09ea18ad5.slice/crio-8fc1b6455bd767871b8407383bcfe6d270ce259a418ba19568c51cc4b6c0e4a6 WatchSource:0}: Error finding container 8fc1b6455bd767871b8407383bcfe6d270ce259a418ba19568c51cc4b6c0e4a6: Status 404 returned error can't find the container with id 8fc1b6455bd767871b8407383bcfe6d270ce259a418ba19568c51cc4b6c0e4a6 Sep 08 16:52:46.064915 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.064880 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k592f\" (UniqueName: \"kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-kube-api-access-k592f\") pod \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " Sep 08 16:52:46.065015 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.064935 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-image-registry-private-configuration\") pod \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " Sep 08 16:52:46.065015 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.064974 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-ca-trust-extracted\") pod \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " Sep 08 16:52:46.065015 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.065001 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-installation-pull-secrets\") pod \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " Sep 08 16:52:46.065182 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.065037 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-bound-sa-token\") pod \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " Sep 08 16:52:46.065182 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.065067 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-trusted-ca\") pod \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " Sep 08 16:52:46.065182 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.065142 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-certificates\") pod \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " Sep 08 16:52:46.065182 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.065175 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls\") pod \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\" (UID: \"4ec136d4-4af2-4a3b-9462-c21f278d4b4a\") " Sep 08 16:52:46.065698 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.065667 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4ec136d4-4af2-4a3b-9462-c21f278d4b4a" (UID: "4ec136d4-4af2-4a3b-9462-c21f278d4b4a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:52:46.066076 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.066011 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4ec136d4-4af2-4a3b-9462-c21f278d4b4a" (UID: "4ec136d4-4af2-4a3b-9462-c21f278d4b4a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:52:46.067960 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.067929 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4ec136d4-4af2-4a3b-9462-c21f278d4b4a" (UID: "4ec136d4-4af2-4a3b-9462-c21f278d4b4a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:52:46.068539 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.068408 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-kube-api-access-k592f" (OuterVolumeSpecName: "kube-api-access-k592f") pod "4ec136d4-4af2-4a3b-9462-c21f278d4b4a" (UID: "4ec136d4-4af2-4a3b-9462-c21f278d4b4a"). InnerVolumeSpecName "kube-api-access-k592f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:52:46.068646 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.068557 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4ec136d4-4af2-4a3b-9462-c21f278d4b4a" (UID: "4ec136d4-4af2-4a3b-9462-c21f278d4b4a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:52:46.069898 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.069870 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4ec136d4-4af2-4a3b-9462-c21f278d4b4a" (UID: "4ec136d4-4af2-4a3b-9462-c21f278d4b4a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:52:46.070160 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.070140 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4ec136d4-4af2-4a3b-9462-c21f278d4b4a" (UID: "4ec136d4-4af2-4a3b-9462-c21f278d4b4a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:52:46.075373 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.075349 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4ec136d4-4af2-4a3b-9462-c21f278d4b4a" (UID: "4ec136d4-4af2-4a3b-9462-c21f278d4b4a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:52:46.166775 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.166740 2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-certificates\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:52:46.166775 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.166776 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-registry-tls\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:52:46.166956 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.166792 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k592f\" (UniqueName: \"kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-kube-api-access-k592f\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:52:46.166956 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.166807 2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-image-registry-private-configuration\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:52:46.166956 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.166817 2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-ca-trust-extracted\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:52:46.166956 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.166826 2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-installation-pull-secrets\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:52:46.166956 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.166840 2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-bound-sa-token\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:52:46.166956 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.166851 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4ec136d4-4af2-4a3b-9462-c21f278d4b4a-trusted-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:52:46.346945 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.346851 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c99c5f9b6-24b2b" event={"ID":"9f4a17c9-7236-4a88-9e26-c99bf5a58569","Type":"ContainerStarted","Data":"7f60ff42a3212ea7944fe32465cb0eb1b788d7de9b4a92ae263bc12d9b7712d5"} Sep 08 16:52:46.348801 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.348770 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7785b5978d-qd69v" event={"ID":"7df39976-8653-457c-9eae-d4d09ea18ad5","Type":"ContainerStarted","Data":"302d1fbd0471755cefe2e15f8e3af67b58faab23e3dab7e2f49ae3551b7cdab6"} Sep 08 16:52:46.348945 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.348820 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7785b5978d-qd69v" event={"ID":"7df39976-8653-457c-9eae-d4d09ea18ad5","Type":"ContainerStarted","Data":"8fc1b6455bd767871b8407383bcfe6d270ce259a418ba19568c51cc4b6c0e4a6"} Sep 08 16:52:46.350149 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.350126 2565 generic.go:356] "Generic (PLEG): container finished" podID="4ec136d4-4af2-4a3b-9462-c21f278d4b4a" containerID="46e5fe18cb242dff61b71e7632ea708f4ee63b7f3b769f58d939c21fa13b9db8" exitCode=0 Sep 08 16:52:46.350264 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.350152 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" event={"ID":"4ec136d4-4af2-4a3b-9462-c21f278d4b4a","Type":"ContainerDied","Data":"46e5fe18cb242dff61b71e7632ea708f4ee63b7f3b769f58d939c21fa13b9db8"} Sep 08 16:52:46.350264 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.350197 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" event={"ID":"4ec136d4-4af2-4a3b-9462-c21f278d4b4a","Type":"ContainerDied","Data":"08e719bfa0eed97e4a7237c6028303d0f4252e408fa88de7ef512e0b1db051f5"} Sep 08 16:52:46.350264 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.350212 2565 scope.go:117] "RemoveContainer" containerID="46e5fe18cb242dff61b71e7632ea708f4ee63b7f3b769f58d939c21fa13b9db8" Sep 08 16:52:46.350393 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.350171 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c8ff9cf7-wh72j" Sep 08 16:52:46.359816 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.359694 2565 scope.go:117] "RemoveContainer" containerID="46e5fe18cb242dff61b71e7632ea708f4ee63b7f3b769f58d939c21fa13b9db8" Sep 08 16:52:46.360014 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:52:46.359982 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46e5fe18cb242dff61b71e7632ea708f4ee63b7f3b769f58d939c21fa13b9db8\": container with ID starting with 46e5fe18cb242dff61b71e7632ea708f4ee63b7f3b769f58d939c21fa13b9db8 not found: ID does not exist" containerID="46e5fe18cb242dff61b71e7632ea708f4ee63b7f3b769f58d939c21fa13b9db8" Sep 08 16:52:46.360125 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.360023 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e5fe18cb242dff61b71e7632ea708f4ee63b7f3b769f58d939c21fa13b9db8"} err="failed to get container status \"46e5fe18cb242dff61b71e7632ea708f4ee63b7f3b769f58d939c21fa13b9db8\": rpc error: code = NotFound desc = could not find container \"46e5fe18cb242dff61b71e7632ea708f4ee63b7f3b769f58d939c21fa13b9db8\": container with ID starting with 46e5fe18cb242dff61b71e7632ea708f4ee63b7f3b769f58d939c21fa13b9db8 not found: ID does not exist" Sep 08 16:52:46.387836 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.387466 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c99c5f9b6-24b2b" podStartSLOduration=8.355501473 podStartE2EDuration="12.387448892s" totalImagesPullingTime="4.031947419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:46.386599008 +0000 UTC m=+129.123430124" watchObservedRunningTime="2026-09-08 16:52:46.387448892 +0000 UTC m=+129.124280011" Sep 08 16:52:46.407894 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.407866 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69c8ff9cf7-wh72j"] Sep 08 16:52:46.412299 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.412276 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69c8ff9cf7-wh72j"] Sep 08 16:52:46.432085 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.432045 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7785b5978d-qd69v" podStartSLOduration=1.432031885 podStartE2EDuration="1.432031885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:46.430814616 +0000 UTC m=+129.167645733" watchObservedRunningTime="2026-09-08 16:52:46.432031885 +0000 UTC m=+129.168863001" Sep 08 16:52:46.670309 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.670281 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs\") pod \"network-metrics-daemon-qbb9j\" (UID: \"5d001365-79a1-4fec-bcf8-3abf7309f1ff\") " pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:52:46.672953 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.672924 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d001365-79a1-4fec-bcf8-3abf7309f1ff-metrics-certs\") pod \"network-metrics-daemon-qbb9j\" (UID: \"5d001365-79a1-4fec-bcf8-3abf7309f1ff\") " pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:52:46.768728 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.768704 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wrtqz\"" Sep 08 16:52:46.776922 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:46.776896 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb9j" Sep 08 16:52:47.092739 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:47.092713 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qbb9j"] Sep 08 16:52:47.096365 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:52:47.096335 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d001365_79a1_4fec_bcf8_3abf7309f1ff.slice/crio-08c8fcb12591bceb8dcc1aa5a039c36e4f0c72b5947b9ab574e16ea475a455dc WatchSource:0}: Error finding container 08c8fcb12591bceb8dcc1aa5a039c36e4f0c72b5947b9ab574e16ea475a455dc: Status 404 returned error can't find the container with id 08c8fcb12591bceb8dcc1aa5a039c36e4f0c72b5947b9ab574e16ea475a455dc Sep 08 16:52:47.356853 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:47.356759 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qbb9j" event={"ID":"5d001365-79a1-4fec-bcf8-3abf7309f1ff","Type":"ContainerStarted","Data":"08c8fcb12591bceb8dcc1aa5a039c36e4f0c72b5947b9ab574e16ea475a455dc"} Sep 08 16:52:47.358640 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:47.358605 2565 generic.go:356] "Generic (PLEG): container finished" podID="c1c8b5c6-12f4-451f-93ac-7e1878b355be" containerID="b75f8026f871af96fe3fd75d7a6f4cf4f8327c9d39ebb603fbaea99852668134" exitCode=0 Sep 08 16:52:47.358782 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:47.358684 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" event={"ID":"c1c8b5c6-12f4-451f-93ac-7e1878b355be","Type":"ContainerDied","Data":"b75f8026f871af96fe3fd75d7a6f4cf4f8327c9d39ebb603fbaea99852668134"} Sep 08 16:52:47.359094 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:47.359069 2565 scope.go:117] "RemoveContainer" containerID="b75f8026f871af96fe3fd75d7a6f4cf4f8327c9d39ebb603fbaea99852668134" Sep 08 16:52:47.761474 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:47.761442 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ec136d4-4af2-4a3b-9462-c21f278d4b4a" path="/var/lib/kubelet/pods/4ec136d4-4af2-4a3b-9462-c21f278d4b4a/volumes" Sep 08 16:52:48.366601 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:48.366556 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-dld4j" event={"ID":"c1c8b5c6-12f4-451f-93ac-7e1878b355be","Type":"ContainerStarted","Data":"29126ff5204acdfc6a8fb4d6216a12b3d77815750e82b4e412e87b8ea20f36b1"} Sep 08 16:52:49.373728 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:49.373640 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qbb9j" event={"ID":"5d001365-79a1-4fec-bcf8-3abf7309f1ff","Type":"ContainerStarted","Data":"44be816425346560cb50d8ce5ec68aff8b3809fe05c27bcaf9396dfd3dac9a6a"} Sep 08 16:52:49.373728 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:49.373704 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qbb9j" event={"ID":"5d001365-79a1-4fec-bcf8-3abf7309f1ff","Type":"ContainerStarted","Data":"6295bace958eca02a9ede753315def219c080133b60643f60b1af8312f5c79fb"} Sep 08 16:52:49.399043 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:49.398985 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qbb9j" podStartSLOduration=129.703950799 podStartE2EDuration="2m11.398965875s" totalImagesPullingTime="1.695015076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:49.398390397 +0000 UTC m=+132.135221514" watchObservedRunningTime="2026-09-08 16:52:49.398965875 +0000 UTC m=+132.135796992" Sep 08 16:52:49.552532 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:49.552499 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:49.684606 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:49.684567 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:52.223852 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:52.223818 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:52.318071 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:52.318033 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:53.359679 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:53.359642 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:53.446073 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:53.446040 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:54.692910 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:54.692877 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:54.693356 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:54.692940 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:54.697417 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:54.697391 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:55.401817 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:55.401789 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:52:55.677341 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:55.677309 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:55.677540 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:55.677366 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:55.682301 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:55.682278 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:55.953663 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:55.953586 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:56.031763 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:56.031729 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:56.405638 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:56.405560 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:52:56.460807 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:56.460778 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c99c5f9b6-24b2b"] Sep 08 16:52:56.784853 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:56.784816 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:56.892988 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:56.892957 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:57.555519 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:57.555489 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:57.632667 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:57.632639 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:58.775960 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:58.775930 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:58.859015 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:58.858981 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:59.177556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:59.177527 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:52:59.281527 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:52:59.281497 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:53:02.425722 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:02.425687 2565 generic.go:356] "Generic (PLEG): container finished" podID="706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5" containerID="bf1e3d26843a4f64e66c42ed0dd3f601b85b77d6114cefb0152ff2bbd7eed617" exitCode=0 Sep 08 16:53:02.426204 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:02.425762 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t48r4" event={"ID":"706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5","Type":"ContainerDied","Data":"bf1e3d26843a4f64e66c42ed0dd3f601b85b77d6114cefb0152ff2bbd7eed617"} Sep 08 16:53:02.426204 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:02.426064 2565 scope.go:117] "RemoveContainer" containerID="bf1e3d26843a4f64e66c42ed0dd3f601b85b77d6114cefb0152ff2bbd7eed617" Sep 08 16:53:03.431619 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:03.431583 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-t48r4" event={"ID":"706c56f3-ffc9-46a5-8bcd-d7a45ecb1ad5","Type":"ContainerStarted","Data":"24e40315a2ea7c832e14dbaaa069d9b8c5bc25f70f7f11c1b04613f5f8ddf8c5"} Sep 08 16:53:10.959666 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:10.959629 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:53:11.060768 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:11.060735 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:53:22.424677 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:22.424628 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c99c5f9b6-24b2b" podUID="9f4a17c9-7236-4a88-9e26-c99bf5a58569" containerName="console" containerID="cri-o://7f60ff42a3212ea7944fe32465cb0eb1b788d7de9b4a92ae263bc12d9b7712d5" gracePeriod=15 Sep 08 16:53:22.702909 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:22.702849 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:53:22.735933 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:22.735898 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b729s\" (UniqueName: \"kubernetes.io/projected/9f4a17c9-7236-4a88-9e26-c99bf5a58569-kube-api-access-b729s\") pod \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " Sep 08 16:53:22.735933 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:22.735935 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f4a17c9-7236-4a88-9e26-c99bf5a58569-console-config\") pod \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " Sep 08 16:53:22.736110 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:22.735967 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f4a17c9-7236-4a88-9e26-c99bf5a58569-console-oauth-config\") pod \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " Sep 08 16:53:22.736110 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:22.735987 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f4a17c9-7236-4a88-9e26-c99bf5a58569-console-serving-cert\") pod \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " Sep 08 16:53:22.736110 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:22.736009 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f4a17c9-7236-4a88-9e26-c99bf5a58569-oauth-serving-cert\") pod \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " Sep 08 16:53:22.736110 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:22.736047 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f4a17c9-7236-4a88-9e26-c99bf5a58569-service-ca\") pod \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\" (UID: \"9f4a17c9-7236-4a88-9e26-c99bf5a58569\") " Sep 08 16:53:22.736507 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:22.736479 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f4a17c9-7236-4a88-9e26-c99bf5a58569-console-config" (OuterVolumeSpecName: "console-config") pod "9f4a17c9-7236-4a88-9e26-c99bf5a58569" (UID: "9f4a17c9-7236-4a88-9e26-c99bf5a58569"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:53:22.736609 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:22.736479 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f4a17c9-7236-4a88-9e26-c99bf5a58569-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9f4a17c9-7236-4a88-9e26-c99bf5a58569" (UID: "9f4a17c9-7236-4a88-9e26-c99bf5a58569"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:53:22.736609 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:22.736519 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f4a17c9-7236-4a88-9e26-c99bf5a58569-service-ca" (OuterVolumeSpecName: "service-ca") pod "9f4a17c9-7236-4a88-9e26-c99bf5a58569" (UID: "9f4a17c9-7236-4a88-9e26-c99bf5a58569"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:53:22.738253 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:22.738222 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f4a17c9-7236-4a88-9e26-c99bf5a58569-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9f4a17c9-7236-4a88-9e26-c99bf5a58569" (UID: "9f4a17c9-7236-4a88-9e26-c99bf5a58569"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:53:22.738253 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:22.738240 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f4a17c9-7236-4a88-9e26-c99bf5a58569-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9f4a17c9-7236-4a88-9e26-c99bf5a58569" (UID: "9f4a17c9-7236-4a88-9e26-c99bf5a58569"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:53:22.738492 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:22.738474 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4a17c9-7236-4a88-9e26-c99bf5a58569-kube-api-access-b729s" (OuterVolumeSpecName: "kube-api-access-b729s") pod "9f4a17c9-7236-4a88-9e26-c99bf5a58569" (UID: "9f4a17c9-7236-4a88-9e26-c99bf5a58569"). InnerVolumeSpecName "kube-api-access-b729s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:53:22.838465 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:22.838439 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b729s\" (UniqueName: \"kubernetes.io/projected/9f4a17c9-7236-4a88-9e26-c99bf5a58569-kube-api-access-b729s\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:53:22.838465 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:22.838464 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f4a17c9-7236-4a88-9e26-c99bf5a58569-console-config\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:53:22.838592 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:22.838474 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f4a17c9-7236-4a88-9e26-c99bf5a58569-console-oauth-config\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:53:22.838592 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:22.838483 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f4a17c9-7236-4a88-9e26-c99bf5a58569-console-serving-cert\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:53:22.838592 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:22.838492 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f4a17c9-7236-4a88-9e26-c99bf5a58569-oauth-serving-cert\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:53:22.838592 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:22.838500 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f4a17c9-7236-4a88-9e26-c99bf5a58569-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:53:23.506967 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:23.506931 2565 generic.go:356] "Generic (PLEG): container finished" podID="9f4a17c9-7236-4a88-9e26-c99bf5a58569" containerID="7f60ff42a3212ea7944fe32465cb0eb1b788d7de9b4a92ae263bc12d9b7712d5" exitCode=0 Sep 08 16:53:23.507343 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:23.506987 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c99c5f9b6-24b2b" Sep 08 16:53:23.507343 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:23.506989 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c99c5f9b6-24b2b" event={"ID":"9f4a17c9-7236-4a88-9e26-c99bf5a58569","Type":"ContainerDied","Data":"7f60ff42a3212ea7944fe32465cb0eb1b788d7de9b4a92ae263bc12d9b7712d5"} Sep 08 16:53:23.507343 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:23.507126 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c99c5f9b6-24b2b" event={"ID":"9f4a17c9-7236-4a88-9e26-c99bf5a58569","Type":"ContainerDied","Data":"ae1c219c3a7c672a225002f3f39f66a09dbb08ec963890803da7d16148456fd2"} Sep 08 16:53:23.507343 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:23.507148 2565 scope.go:117] "RemoveContainer" containerID="7f60ff42a3212ea7944fe32465cb0eb1b788d7de9b4a92ae263bc12d9b7712d5" Sep 08 16:53:23.516241 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:23.516219 2565 scope.go:117] "RemoveContainer" containerID="7f60ff42a3212ea7944fe32465cb0eb1b788d7de9b4a92ae263bc12d9b7712d5" Sep 08 16:53:23.516518 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:53:23.516498 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f60ff42a3212ea7944fe32465cb0eb1b788d7de9b4a92ae263bc12d9b7712d5\": container with ID starting with 7f60ff42a3212ea7944fe32465cb0eb1b788d7de9b4a92ae263bc12d9b7712d5 not found: ID does not exist" containerID="7f60ff42a3212ea7944fe32465cb0eb1b788d7de9b4a92ae263bc12d9b7712d5" Sep 08 16:53:23.516580 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:23.516528 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f60ff42a3212ea7944fe32465cb0eb1b788d7de9b4a92ae263bc12d9b7712d5"} err="failed to get container status \"7f60ff42a3212ea7944fe32465cb0eb1b788d7de9b4a92ae263bc12d9b7712d5\": rpc error: code = NotFound desc = could not find container \"7f60ff42a3212ea7944fe32465cb0eb1b788d7de9b4a92ae263bc12d9b7712d5\": container with ID starting with 7f60ff42a3212ea7944fe32465cb0eb1b788d7de9b4a92ae263bc12d9b7712d5 not found: ID does not exist" Sep 08 16:53:23.530379 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:23.530348 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c99c5f9b6-24b2b"] Sep 08 16:53:23.536952 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:23.536932 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c99c5f9b6-24b2b"] Sep 08 16:53:23.758834 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:23.758782 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f4a17c9-7236-4a88-9e26-c99bf5a58569" path="/var/lib/kubelet/pods/9f4a17c9-7236-4a88-9e26-c99bf5a58569/volumes" Sep 08 16:53:27.451052 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:27.451018 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:53:27.540470 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:27.540441 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:53:39.410148 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:39.410109 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:53:39.506553 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:39.506516 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:53:41.264690 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.264649 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-89b7c5f89-c7qk9"] Sep 08 16:53:41.265264 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.265242 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ec136d4-4af2-4a3b-9462-c21f278d4b4a" containerName="registry" Sep 08 16:53:41.265349 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.265277 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ec136d4-4af2-4a3b-9462-c21f278d4b4a" containerName="registry" Sep 08 16:53:41.265349 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.265295 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f4a17c9-7236-4a88-9e26-c99bf5a58569" containerName="console" Sep 08 16:53:41.265349 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.265305 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f4a17c9-7236-4a88-9e26-c99bf5a58569" containerName="console" Sep 08 16:53:41.265544 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.265474 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ec136d4-4af2-4a3b-9462-c21f278d4b4a" containerName="registry" Sep 08 16:53:41.265544 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.265487 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f4a17c9-7236-4a88-9e26-c99bf5a58569" containerName="console" Sep 08 16:53:41.270392 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.270363 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.293754 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.293726 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-89b7c5f89-c7qk9"] Sep 08 16:53:41.369383 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.369346 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-oauth-serving-cert\") pod \"console-89b7c5f89-c7qk9\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.369572 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.369450 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ae831f9-ffb0-4484-9b05-d888780464d1-console-serving-cert\") pod \"console-89b7c5f89-c7qk9\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.369572 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.369498 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-trusted-ca-bundle\") pod \"console-89b7c5f89-c7qk9\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.369687 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.369569 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg2gp\" (UniqueName: \"kubernetes.io/projected/3ae831f9-ffb0-4484-9b05-d888780464d1-kube-api-access-jg2gp\") pod \"console-89b7c5f89-c7qk9\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.369687 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.369605 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-service-ca\") pod \"console-89b7c5f89-c7qk9\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.369687 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.369645 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ae831f9-ffb0-4484-9b05-d888780464d1-console-oauth-config\") pod \"console-89b7c5f89-c7qk9\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.369687 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.369670 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-console-config\") pod \"console-89b7c5f89-c7qk9\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.370313 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.370284 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:53:41.470655 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.470629 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-service-ca\") pod \"console-89b7c5f89-c7qk9\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.470781 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.470669 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ae831f9-ffb0-4484-9b05-d888780464d1-console-oauth-config\") pod \"console-89b7c5f89-c7qk9\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.470781 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.470685 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-console-config\") pod \"console-89b7c5f89-c7qk9\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.470903 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.470885 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-oauth-serving-cert\") pod \"console-89b7c5f89-c7qk9\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.470962 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.470923 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ae831f9-ffb0-4484-9b05-d888780464d1-console-serving-cert\") pod \"console-89b7c5f89-c7qk9\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.470962 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.470949 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-trusted-ca-bundle\") pod \"console-89b7c5f89-c7qk9\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.471058 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.470982 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg2gp\" (UniqueName: \"kubernetes.io/projected/3ae831f9-ffb0-4484-9b05-d888780464d1-kube-api-access-jg2gp\") pod \"console-89b7c5f89-c7qk9\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.471599 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.471565 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-service-ca\") pod \"console-89b7c5f89-c7qk9\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.471723 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.471565 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-console-config\") pod \"console-89b7c5f89-c7qk9\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.471723 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.471649 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-oauth-serving-cert\") pod \"console-89b7c5f89-c7qk9\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.471850 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.471783 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-trusted-ca-bundle\") pod \"console-89b7c5f89-c7qk9\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.473076 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.473057 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ae831f9-ffb0-4484-9b05-d888780464d1-console-oauth-config\") pod \"console-89b7c5f89-c7qk9\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.473347 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.473330 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ae831f9-ffb0-4484-9b05-d888780464d1-console-serving-cert\") pod \"console-89b7c5f89-c7qk9\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.476697 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.476670 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:53:41.480065 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.480041 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg2gp\" (UniqueName: \"kubernetes.io/projected/3ae831f9-ffb0-4484-9b05-d888780464d1-kube-api-access-jg2gp\") pod \"console-89b7c5f89-c7qk9\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.579728 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.579649 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:41.876222 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:41.876155 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-89b7c5f89-c7qk9"] Sep 08 16:53:41.879186 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:53:41.879159 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae831f9_ffb0_4484_9b05_d888780464d1.slice/crio-fa2743446b03915d6ae3ebc9566cd8c7c1ef6b7946ca634955a7a87efb458fa2 WatchSource:0}: Error finding container fa2743446b03915d6ae3ebc9566cd8c7c1ef6b7946ca634955a7a87efb458fa2: Status 404 returned error can't find the container with id fa2743446b03915d6ae3ebc9566cd8c7c1ef6b7946ca634955a7a87efb458fa2 Sep 08 16:53:42.582853 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:42.582816 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-89b7c5f89-c7qk9" event={"ID":"3ae831f9-ffb0-4484-9b05-d888780464d1","Type":"ContainerStarted","Data":"d8cd4455900a9c068462f35bf6efeb80b507211798b52802ec6188e656832578"} Sep 08 16:53:42.583215 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:42.582870 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-89b7c5f89-c7qk9" event={"ID":"3ae831f9-ffb0-4484-9b05-d888780464d1","Type":"ContainerStarted","Data":"fa2743446b03915d6ae3ebc9566cd8c7c1ef6b7946ca634955a7a87efb458fa2"} Sep 08 16:53:42.608876 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:42.608833 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-89b7c5f89-c7qk9" podStartSLOduration=1.60881994 podStartE2EDuration="1.60881994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:53:42.606653517 +0000 UTC m=+185.343484632" watchObservedRunningTime="2026-09-08 16:53:42.60881994 +0000 UTC m=+185.345651055" Sep 08 16:53:51.580402 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:51.580362 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:51.580881 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:51.580439 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:51.585202 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:51.585178 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:51.622294 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:51.622264 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:53:51.689680 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:51.689648 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7785b5978d-qd69v"] Sep 08 16:53:51.919155 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:51.919117 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:53:52.018378 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:52.018346 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:53:55.989365 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:55.989323 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:53:56.133664 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:53:56.133633 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:54:16.709641 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:16.709601 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7785b5978d-qd69v" podUID="7df39976-8653-457c-9eae-d4d09ea18ad5" containerName="console" containerID="cri-o://302d1fbd0471755cefe2e15f8e3af67b58faab23e3dab7e2f49ae3551b7cdab6" gracePeriod=15 Sep 08 16:54:16.949198 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:16.949175 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:54:17.041403 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.041312 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-trusted-ca-bundle\") pod \"7df39976-8653-457c-9eae-d4d09ea18ad5\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " Sep 08 16:54:17.041403 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.041353 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4vq7\" (UniqueName: \"kubernetes.io/projected/7df39976-8653-457c-9eae-d4d09ea18ad5-kube-api-access-j4vq7\") pod \"7df39976-8653-457c-9eae-d4d09ea18ad5\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " Sep 08 16:54:17.041403 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.041376 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-console-config\") pod \"7df39976-8653-457c-9eae-d4d09ea18ad5\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " Sep 08 16:54:17.041691 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.041412 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7df39976-8653-457c-9eae-d4d09ea18ad5-console-oauth-config\") pod \"7df39976-8653-457c-9eae-d4d09ea18ad5\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " Sep 08 16:54:17.041691 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.041465 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7df39976-8653-457c-9eae-d4d09ea18ad5-console-serving-cert\") pod \"7df39976-8653-457c-9eae-d4d09ea18ad5\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " Sep 08 16:54:17.041691 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.041517 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-oauth-serving-cert\") pod \"7df39976-8653-457c-9eae-d4d09ea18ad5\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " Sep 08 16:54:17.041691 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.041543 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-service-ca\") pod \"7df39976-8653-457c-9eae-d4d09ea18ad5\" (UID: \"7df39976-8653-457c-9eae-d4d09ea18ad5\") " Sep 08 16:54:17.041884 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.041846 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-console-config" (OuterVolumeSpecName: "console-config") pod "7df39976-8653-457c-9eae-d4d09ea18ad5" (UID: "7df39976-8653-457c-9eae-d4d09ea18ad5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:54:17.041982 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.041951 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7df39976-8653-457c-9eae-d4d09ea18ad5" (UID: "7df39976-8653-457c-9eae-d4d09ea18ad5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:54:17.042044 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.041995 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7df39976-8653-457c-9eae-d4d09ea18ad5" (UID: "7df39976-8653-457c-9eae-d4d09ea18ad5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:54:17.042096 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.042058 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-service-ca" (OuterVolumeSpecName: "service-ca") pod "7df39976-8653-457c-9eae-d4d09ea18ad5" (UID: "7df39976-8653-457c-9eae-d4d09ea18ad5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:54:17.043588 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.043566 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df39976-8653-457c-9eae-d4d09ea18ad5-kube-api-access-j4vq7" (OuterVolumeSpecName: "kube-api-access-j4vq7") pod "7df39976-8653-457c-9eae-d4d09ea18ad5" (UID: "7df39976-8653-457c-9eae-d4d09ea18ad5"). InnerVolumeSpecName "kube-api-access-j4vq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:54:17.043672 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.043570 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7df39976-8653-457c-9eae-d4d09ea18ad5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7df39976-8653-457c-9eae-d4d09ea18ad5" (UID: "7df39976-8653-457c-9eae-d4d09ea18ad5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:54:17.043672 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.043601 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7df39976-8653-457c-9eae-d4d09ea18ad5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7df39976-8653-457c-9eae-d4d09ea18ad5" (UID: "7df39976-8653-457c-9eae-d4d09ea18ad5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:54:17.142311 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.142258 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-trusted-ca-bundle\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:54:17.142311 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.142305 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4vq7\" (UniqueName: \"kubernetes.io/projected/7df39976-8653-457c-9eae-d4d09ea18ad5-kube-api-access-j4vq7\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:54:17.142311 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.142316 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-console-config\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:54:17.142311 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.142326 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7df39976-8653-457c-9eae-d4d09ea18ad5-console-oauth-config\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:54:17.142616 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.142340 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7df39976-8653-457c-9eae-d4d09ea18ad5-console-serving-cert\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:54:17.142616 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.142351 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-oauth-serving-cert\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:54:17.142616 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.142359 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7df39976-8653-457c-9eae-d4d09ea18ad5-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:54:17.713250 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.713206 2565 generic.go:356] "Generic (PLEG): container finished" podID="7df39976-8653-457c-9eae-d4d09ea18ad5" containerID="302d1fbd0471755cefe2e15f8e3af67b58faab23e3dab7e2f49ae3551b7cdab6" exitCode=0 Sep 08 16:54:17.713250 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.713257 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7785b5978d-qd69v" Sep 08 16:54:17.713710 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.713289 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7785b5978d-qd69v" event={"ID":"7df39976-8653-457c-9eae-d4d09ea18ad5","Type":"ContainerDied","Data":"302d1fbd0471755cefe2e15f8e3af67b58faab23e3dab7e2f49ae3551b7cdab6"} Sep 08 16:54:17.713710 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.713339 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7785b5978d-qd69v" event={"ID":"7df39976-8653-457c-9eae-d4d09ea18ad5","Type":"ContainerDied","Data":"8fc1b6455bd767871b8407383bcfe6d270ce259a418ba19568c51cc4b6c0e4a6"} Sep 08 16:54:17.713710 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.713364 2565 scope.go:117] "RemoveContainer" containerID="302d1fbd0471755cefe2e15f8e3af67b58faab23e3dab7e2f49ae3551b7cdab6" Sep 08 16:54:17.722340 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.722306 2565 scope.go:117] "RemoveContainer" containerID="302d1fbd0471755cefe2e15f8e3af67b58faab23e3dab7e2f49ae3551b7cdab6" Sep 08 16:54:17.722641 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:54:17.722622 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"302d1fbd0471755cefe2e15f8e3af67b58faab23e3dab7e2f49ae3551b7cdab6\": container with ID starting with 302d1fbd0471755cefe2e15f8e3af67b58faab23e3dab7e2f49ae3551b7cdab6 not found: ID does not exist" containerID="302d1fbd0471755cefe2e15f8e3af67b58faab23e3dab7e2f49ae3551b7cdab6" Sep 08 16:54:17.722712 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.722650 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"302d1fbd0471755cefe2e15f8e3af67b58faab23e3dab7e2f49ae3551b7cdab6"} err="failed to get container status \"302d1fbd0471755cefe2e15f8e3af67b58faab23e3dab7e2f49ae3551b7cdab6\": rpc error: code = NotFound desc = could not find container \"302d1fbd0471755cefe2e15f8e3af67b58faab23e3dab7e2f49ae3551b7cdab6\": container with ID starting with 302d1fbd0471755cefe2e15f8e3af67b58faab23e3dab7e2f49ae3551b7cdab6 not found: ID does not exist" Sep 08 16:54:17.735130 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.735104 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7785b5978d-qd69v"] Sep 08 16:54:17.740086 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.740061 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7785b5978d-qd69v"] Sep 08 16:54:17.757729 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:17.757705 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7df39976-8653-457c-9eae-d4d09ea18ad5" path="/var/lib/kubelet/pods/7df39976-8653-457c-9eae-d4d09ea18ad5/volumes" Sep 08 16:54:21.734676 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:21.734647 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:54:21.858064 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:21.858029 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:54:23.584062 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:23.584031 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4"] Sep 08 16:54:23.584475 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:23.584294 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7df39976-8653-457c-9eae-d4d09ea18ad5" containerName="console" Sep 08 16:54:23.584475 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:23.584305 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7df39976-8653-457c-9eae-d4d09ea18ad5" containerName="console" Sep 08 16:54:23.584475 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:23.584360 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7df39976-8653-457c-9eae-d4d09ea18ad5" containerName="console" Sep 08 16:54:23.588645 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:23.588627 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4" Sep 08 16:54:23.593623 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:23.593594 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-v8k2r\"" Sep 08 16:54:23.593736 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:23.593708 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 16:54:23.594160 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:23.594144 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 16:54:23.605640 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:23.605615 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4"] Sep 08 16:54:23.693217 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:23.693188 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6c1dc90-2c21-4897-b163-266f12462b3e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4\" (UID: \"f6c1dc90-2c21-4897-b163-266f12462b3e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4" Sep 08 16:54:23.693344 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:23.693230 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6c1dc90-2c21-4897-b163-266f12462b3e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4\" (UID: \"f6c1dc90-2c21-4897-b163-266f12462b3e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4" Sep 08 16:54:23.693344 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:23.693291 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmmv8\" (UniqueName: \"kubernetes.io/projected/f6c1dc90-2c21-4897-b163-266f12462b3e-kube-api-access-xmmv8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4\" (UID: \"f6c1dc90-2c21-4897-b163-266f12462b3e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4" Sep 08 16:54:23.793670 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:23.793647 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6c1dc90-2c21-4897-b163-266f12462b3e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4\" (UID: \"f6c1dc90-2c21-4897-b163-266f12462b3e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4" Sep 08 16:54:23.793781 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:23.793684 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6c1dc90-2c21-4897-b163-266f12462b3e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4\" (UID: \"f6c1dc90-2c21-4897-b163-266f12462b3e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4" Sep 08 16:54:23.793781 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:23.793719 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmmv8\" (UniqueName: \"kubernetes.io/projected/f6c1dc90-2c21-4897-b163-266f12462b3e-kube-api-access-xmmv8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4\" (UID: \"f6c1dc90-2c21-4897-b163-266f12462b3e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4" Sep 08 16:54:23.793997 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:23.793979 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6c1dc90-2c21-4897-b163-266f12462b3e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4\" (UID: \"f6c1dc90-2c21-4897-b163-266f12462b3e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4" Sep 08 16:54:23.794033 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:23.794007 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6c1dc90-2c21-4897-b163-266f12462b3e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4\" (UID: \"f6c1dc90-2c21-4897-b163-266f12462b3e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4" Sep 08 16:54:23.811396 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:23.811369 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmmv8\" (UniqueName: \"kubernetes.io/projected/f6c1dc90-2c21-4897-b163-266f12462b3e-kube-api-access-xmmv8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4\" (UID: \"f6c1dc90-2c21-4897-b163-266f12462b3e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4" Sep 08 16:54:23.897018 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:23.896960 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4" Sep 08 16:54:24.172905 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:24.172872 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4"] Sep 08 16:54:24.176468 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:54:24.176440 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c1dc90_2c21_4897_b163_266f12462b3e.slice/crio-2a8a9a100e2d73dd2cdff1383fe75c673ab23a95378eedd3bfd8876b885cfcb5 WatchSource:0}: Error finding container 2a8a9a100e2d73dd2cdff1383fe75c673ab23a95378eedd3bfd8876b885cfcb5: Status 404 returned error can't find the container with id 2a8a9a100e2d73dd2cdff1383fe75c673ab23a95378eedd3bfd8876b885cfcb5 Sep 08 16:54:24.741221 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:24.741183 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4" event={"ID":"f6c1dc90-2c21-4897-b163-266f12462b3e","Type":"ContainerStarted","Data":"2a8a9a100e2d73dd2cdff1383fe75c673ab23a95378eedd3bfd8876b885cfcb5"} Sep 08 16:54:30.766433 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:30.766380 2565 generic.go:356] "Generic (PLEG): container finished" podID="f6c1dc90-2c21-4897-b163-266f12462b3e" containerID="ef52296a6b5ab414019eea41aabb3970de771a17d516cec64d91edc88da25943" exitCode=0 Sep 08 16:54:30.766808 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:30.766457 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4" event={"ID":"f6c1dc90-2c21-4897-b163-266f12462b3e","Type":"ContainerDied","Data":"ef52296a6b5ab414019eea41aabb3970de771a17d516cec64d91edc88da25943"} Sep 08 16:54:33.780074 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:33.780031 2565 generic.go:356] "Generic (PLEG): container finished" podID="f6c1dc90-2c21-4897-b163-266f12462b3e" containerID="ba2916b5160c95e87c62857d6b0af8d424e5ac6391656e5777a8ff0576a30ad7" exitCode=0 Sep 08 16:54:33.780532 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:33.780087 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4" event={"ID":"f6c1dc90-2c21-4897-b163-266f12462b3e","Type":"ContainerDied","Data":"ba2916b5160c95e87c62857d6b0af8d424e5ac6391656e5777a8ff0576a30ad7"} Sep 08 16:54:40.811692 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:40.811651 2565 generic.go:356] "Generic (PLEG): container finished" podID="f6c1dc90-2c21-4897-b163-266f12462b3e" containerID="9fff579105ac305611bc3e032e4593bd2756be26f6278d5372829c06bbd19ca1" exitCode=0 Sep 08 16:54:40.812108 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:40.811713 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4" event={"ID":"f6c1dc90-2c21-4897-b163-266f12462b3e","Type":"ContainerDied","Data":"9fff579105ac305611bc3e032e4593bd2756be26f6278d5372829c06bbd19ca1"} Sep 08 16:54:41.935952 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:41.935928 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4" Sep 08 16:54:41.937461 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:41.937417 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmmv8\" (UniqueName: \"kubernetes.io/projected/f6c1dc90-2c21-4897-b163-266f12462b3e-kube-api-access-xmmv8\") pod \"f6c1dc90-2c21-4897-b163-266f12462b3e\" (UID: \"f6c1dc90-2c21-4897-b163-266f12462b3e\") " Sep 08 16:54:41.937596 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:41.937576 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6c1dc90-2c21-4897-b163-266f12462b3e-bundle\") pod \"f6c1dc90-2c21-4897-b163-266f12462b3e\" (UID: \"f6c1dc90-2c21-4897-b163-266f12462b3e\") " Sep 08 16:54:41.937670 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:41.937657 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6c1dc90-2c21-4897-b163-266f12462b3e-util\") pod \"f6c1dc90-2c21-4897-b163-266f12462b3e\" (UID: \"f6c1dc90-2c21-4897-b163-266f12462b3e\") " Sep 08 16:54:41.938178 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:41.938149 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c1dc90-2c21-4897-b163-266f12462b3e-bundle" (OuterVolumeSpecName: "bundle") pod "f6c1dc90-2c21-4897-b163-266f12462b3e" (UID: "f6c1dc90-2c21-4897-b163-266f12462b3e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:54:41.939539 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:41.939510 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c1dc90-2c21-4897-b163-266f12462b3e-kube-api-access-xmmv8" (OuterVolumeSpecName: "kube-api-access-xmmv8") pod "f6c1dc90-2c21-4897-b163-266f12462b3e" (UID: "f6c1dc90-2c21-4897-b163-266f12462b3e"). InnerVolumeSpecName "kube-api-access-xmmv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:54:41.942527 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:41.942505 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c1dc90-2c21-4897-b163-266f12462b3e-util" (OuterVolumeSpecName: "util") pod "f6c1dc90-2c21-4897-b163-266f12462b3e" (UID: "f6c1dc90-2c21-4897-b163-266f12462b3e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:54:42.038376 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:42.038341 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmmv8\" (UniqueName: \"kubernetes.io/projected/f6c1dc90-2c21-4897-b163-266f12462b3e-kube-api-access-xmmv8\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:54:42.038376 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:42.038372 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6c1dc90-2c21-4897-b163-266f12462b3e-bundle\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:54:42.038376 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:42.038382 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6c1dc90-2c21-4897-b163-266f12462b3e-util\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:54:42.820098 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:42.820063 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4" event={"ID":"f6c1dc90-2c21-4897-b163-266f12462b3e","Type":"ContainerDied","Data":"2a8a9a100e2d73dd2cdff1383fe75c673ab23a95378eedd3bfd8876b885cfcb5"} Sep 08 16:54:42.820239 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:42.820110 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a8a9a100e2d73dd2cdff1383fe75c673ab23a95378eedd3bfd8876b885cfcb5" Sep 08 16:54:42.820239 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:42.820079 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kgmt4" Sep 08 16:54:44.911544 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:44.911511 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:54:45.247787 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:45.247741 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:54:47.349419 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:47.349381 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zsdnp"] Sep 08 16:54:47.349802 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:47.349675 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6c1dc90-2c21-4897-b163-266f12462b3e" containerName="extract" Sep 08 16:54:47.349802 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:47.349687 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6c1dc90-2c21-4897-b163-266f12462b3e" containerName="extract" Sep 08 16:54:47.349802 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:47.349700 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6c1dc90-2c21-4897-b163-266f12462b3e" containerName="util" Sep 08 16:54:47.349802 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:47.349706 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6c1dc90-2c21-4897-b163-266f12462b3e" containerName="util" Sep 08 16:54:47.349802 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:47.349712 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6c1dc90-2c21-4897-b163-266f12462b3e" containerName="pull" Sep 08 16:54:47.349802 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:47.349717 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6c1dc90-2c21-4897-b163-266f12462b3e" containerName="pull" Sep 08 16:54:47.349802 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:47.349763 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6c1dc90-2c21-4897-b163-266f12462b3e" containerName="extract" Sep 08 16:54:47.353250 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:47.353234 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zsdnp" Sep 08 16:54:47.357358 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:47.357336 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 08 16:54:47.357500 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:47.357384 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-bm4c8\"" Sep 08 16:54:47.357623 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:47.357609 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:54:47.374447 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:47.374409 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgrqq\" (UniqueName: \"kubernetes.io/projected/a28f2bb7-afd2-4460-8118-1b5c37b9e9f4-kube-api-access-sgrqq\") pod \"cert-manager-operator-controller-manager-55c444bf86-zsdnp\" (UID: \"a28f2bb7-afd2-4460-8118-1b5c37b9e9f4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zsdnp" Sep 08 16:54:47.374579 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:47.374550 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a28f2bb7-afd2-4460-8118-1b5c37b9e9f4-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-zsdnp\" (UID: \"a28f2bb7-afd2-4460-8118-1b5c37b9e9f4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zsdnp" Sep 08 16:54:47.375830 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:47.375808 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zsdnp"] Sep 08 16:54:47.475388 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:47.475356 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgrqq\" (UniqueName: \"kubernetes.io/projected/a28f2bb7-afd2-4460-8118-1b5c37b9e9f4-kube-api-access-sgrqq\") pod \"cert-manager-operator-controller-manager-55c444bf86-zsdnp\" (UID: \"a28f2bb7-afd2-4460-8118-1b5c37b9e9f4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zsdnp" Sep 08 16:54:47.475552 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:47.475442 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a28f2bb7-afd2-4460-8118-1b5c37b9e9f4-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-zsdnp\" (UID: \"a28f2bb7-afd2-4460-8118-1b5c37b9e9f4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zsdnp" Sep 08 16:54:47.475814 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:47.475796 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a28f2bb7-afd2-4460-8118-1b5c37b9e9f4-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-zsdnp\" (UID: \"a28f2bb7-afd2-4460-8118-1b5c37b9e9f4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zsdnp" Sep 08 16:54:47.484957 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:47.484935 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgrqq\" (UniqueName: \"kubernetes.io/projected/a28f2bb7-afd2-4460-8118-1b5c37b9e9f4-kube-api-access-sgrqq\") pod \"cert-manager-operator-controller-manager-55c444bf86-zsdnp\" (UID: \"a28f2bb7-afd2-4460-8118-1b5c37b9e9f4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zsdnp" Sep 08 16:54:47.661815 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:47.661752 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zsdnp" Sep 08 16:54:47.954486 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:47.954456 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zsdnp"] Sep 08 16:54:47.958045 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:54:47.958009 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda28f2bb7_afd2_4460_8118_1b5c37b9e9f4.slice/crio-6857ab4f3cb86031d5167fa9358d7d7284ba7bef412b010cad4175d58dc1aa59 WatchSource:0}: Error finding container 6857ab4f3cb86031d5167fa9358d7d7284ba7bef412b010cad4175d58dc1aa59: Status 404 returned error can't find the container with id 6857ab4f3cb86031d5167fa9358d7d7284ba7bef412b010cad4175d58dc1aa59 Sep 08 16:54:48.846709 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:48.846668 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zsdnp" event={"ID":"a28f2bb7-afd2-4460-8118-1b5c37b9e9f4","Type":"ContainerStarted","Data":"6857ab4f3cb86031d5167fa9358d7d7284ba7bef412b010cad4175d58dc1aa59"} Sep 08 16:54:48.991203 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:48.991173 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:54:49.114219 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:49.114129 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:54:49.852806 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:49.852766 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zsdnp" event={"ID":"a28f2bb7-afd2-4460-8118-1b5c37b9e9f4","Type":"ContainerStarted","Data":"3c772254f50c257a1a88d8c33da10d617e94c7fba10df2a58930fadd8c255110"} Sep 08 16:54:49.938243 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:49.938191 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zsdnp" podStartSLOduration=1.554161926 podStartE2EDuration="2.938168839s" totalImagesPullingTime="1.384006913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:54:49.935383943 +0000 UTC m=+252.672215060" watchObservedRunningTime="2026-09-08 16:54:49.938168839 +0000 UTC m=+252.674999959" Sep 08 16:54:50.483162 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:50.483129 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:54:50.734457 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:50.734332 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:54:51.068232 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:51.068158 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g"] Sep 08 16:54:51.075320 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:51.075294 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g" Sep 08 16:54:51.080197 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:51.080176 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 16:54:51.081287 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:51.081252 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-v8k2r\"" Sep 08 16:54:51.081967 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:51.081849 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 16:54:51.099884 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:51.099854 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g\" (UID: \"5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g" Sep 08 16:54:51.099972 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:51.099914 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g\" (UID: \"5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g" Sep 08 16:54:51.099972 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:51.099955 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k5tg\" (UniqueName: \"kubernetes.io/projected/5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8-kube-api-access-7k5tg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g\" (UID: \"5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g" Sep 08 16:54:51.108752 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:51.108730 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g"] Sep 08 16:54:51.201098 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:51.201070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g\" (UID: \"5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g" Sep 08 16:54:51.201219 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:51.201115 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g\" (UID: \"5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g" Sep 08 16:54:51.201219 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:51.201154 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k5tg\" (UniqueName: \"kubernetes.io/projected/5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8-kube-api-access-7k5tg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g\" (UID: \"5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g" Sep 08 16:54:51.201445 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:51.201409 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g\" (UID: \"5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g" Sep 08 16:54:51.201551 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:51.201534 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g\" (UID: \"5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g" Sep 08 16:54:51.233912 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:51.233884 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k5tg\" (UniqueName: \"kubernetes.io/projected/5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8-kube-api-access-7k5tg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g\" (UID: \"5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g" Sep 08 16:54:51.261029 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:51.261002 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:54:51.384055 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:51.383986 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g" Sep 08 16:54:51.495316 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:51.495284 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:54:51.683405 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:51.683379 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g"] Sep 08 16:54:51.685836 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:54:51.685808 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5daf3fce_61be_4ac7_8da0_1fd0b5e6d7e8.slice/crio-30be553dea90ab408327ccd8fa5dc36f08332a7e75833dde192f9b2b6ee08b20 WatchSource:0}: Error finding container 30be553dea90ab408327ccd8fa5dc36f08332a7e75833dde192f9b2b6ee08b20: Status 404 returned error can't find the container with id 30be553dea90ab408327ccd8fa5dc36f08332a7e75833dde192f9b2b6ee08b20 Sep 08 16:54:51.861656 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:51.861625 2565 generic.go:356] "Generic (PLEG): container finished" podID="5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8" containerID="9e8d70ef452ea22c6cdb4eecc0ff59bce731c92bea5eb7fd80fb7e4382721d81" exitCode=0 Sep 08 16:54:51.861762 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:51.861712 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g" event={"ID":"5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8","Type":"ContainerDied","Data":"9e8d70ef452ea22c6cdb4eecc0ff59bce731c92bea5eb7fd80fb7e4382721d81"} Sep 08 16:54:51.861805 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:51.861761 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g" event={"ID":"5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8","Type":"ContainerStarted","Data":"30be553dea90ab408327ccd8fa5dc36f08332a7e75833dde192f9b2b6ee08b20"} Sep 08 16:54:53.467892 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:53.467861 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:54:53.858964 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:53.858936 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:54:53.873405 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:53.873381 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g" event={"ID":"5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8","Type":"ContainerStarted","Data":"dcce768486551653aeb0d94c1c9fc58101229c4c17caace811902d8f897c95ec"} Sep 08 16:54:54.879224 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:54.879190 2565 generic.go:356] "Generic (PLEG): container finished" podID="5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8" containerID="dcce768486551653aeb0d94c1c9fc58101229c4c17caace811902d8f897c95ec" exitCode=0 Sep 08 16:54:54.879644 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:54.879275 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g" event={"ID":"5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8","Type":"ContainerDied","Data":"dcce768486551653aeb0d94c1c9fc58101229c4c17caace811902d8f897c95ec"} Sep 08 16:54:55.715467 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:55.715412 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:54:55.816226 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:55.816198 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:54:55.890351 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:55.890312 2565 generic.go:356] "Generic (PLEG): container finished" podID="5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8" containerID="d43c76b098fe60acefbde8e0d67b7e1b97695e020407b0433365b38faaa06f5f" exitCode=0 Sep 08 16:54:55.890743 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:55.890370 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g" event={"ID":"5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8","Type":"ContainerDied","Data":"d43c76b098fe60acefbde8e0d67b7e1b97695e020407b0433365b38faaa06f5f"} Sep 08 16:54:57.012146 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:57.012121 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g" Sep 08 16:54:57.039814 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:57.039785 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7k5tg\" (UniqueName: \"kubernetes.io/projected/5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8-kube-api-access-7k5tg\") pod \"5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8\" (UID: \"5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8\") " Sep 08 16:54:57.039962 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:57.039845 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8-util\") pod \"5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8\" (UID: \"5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8\") " Sep 08 16:54:57.040027 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:57.039970 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8-bundle\") pod \"5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8\" (UID: \"5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8\") " Sep 08 16:54:57.040484 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:57.040356 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8-bundle" (OuterVolumeSpecName: "bundle") pod "5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8" (UID: "5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:54:57.042137 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:57.042113 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8-kube-api-access-7k5tg" (OuterVolumeSpecName: "kube-api-access-7k5tg") pod "5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8" (UID: "5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8"). InnerVolumeSpecName "kube-api-access-7k5tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:54:57.045737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:57.045714 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8-util" (OuterVolumeSpecName: "util") pod "5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8" (UID: "5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:54:57.140679 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:57.140655 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8-bundle\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:54:57.140679 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:57.140677 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k5tg\" (UniqueName: \"kubernetes.io/projected/5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8-kube-api-access-7k5tg\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:54:57.140809 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:57.140687 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8-util\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:54:57.900943 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:57.900913 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g" Sep 08 16:54:57.900943 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:57.900922 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8b27g" event={"ID":"5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8","Type":"ContainerDied","Data":"30be553dea90ab408327ccd8fa5dc36f08332a7e75833dde192f9b2b6ee08b20"} Sep 08 16:54:57.901166 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:57.900965 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30be553dea90ab408327ccd8fa5dc36f08332a7e75833dde192f9b2b6ee08b20" Sep 08 16:54:58.360292 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:58.360256 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z8225"] Sep 08 16:54:58.360661 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:58.360544 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8" containerName="util" Sep 08 16:54:58.360661 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:58.360555 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8" containerName="util" Sep 08 16:54:58.360661 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:58.360567 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8" containerName="extract" Sep 08 16:54:58.360661 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:58.360572 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8" containerName="extract" Sep 08 16:54:58.360661 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:58.360584 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8" containerName="pull" Sep 08 16:54:58.360661 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:58.360589 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8" containerName="pull" Sep 08 16:54:58.360661 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:58.360633 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5daf3fce-61be-4ac7-8da0-1fd0b5e6d7e8" containerName="extract" Sep 08 16:54:58.364847 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:58.364826 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z8225" Sep 08 16:54:58.367815 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:58.367785 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 08 16:54:58.368482 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:58.368460 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-qd6gt\"" Sep 08 16:54:58.368770 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:58.368754 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 08 16:54:58.381160 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:58.381140 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z8225"] Sep 08 16:54:58.449786 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:58.449760 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjpn2\" (UniqueName: \"kubernetes.io/projected/e1d379a5-61b7-4e77-8f12-e2f3c0a22bef-kube-api-access-kjpn2\") pod \"cert-manager-cainjector-559cdd8c87-z8225\" (UID: \"e1d379a5-61b7-4e77-8f12-e2f3c0a22bef\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z8225" Sep 08 16:54:58.449878 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:58.449794 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1d379a5-61b7-4e77-8f12-e2f3c0a22bef-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z8225\" (UID: \"e1d379a5-61b7-4e77-8f12-e2f3c0a22bef\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z8225" Sep 08 16:54:58.551004 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:58.550979 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjpn2\" (UniqueName: \"kubernetes.io/projected/e1d379a5-61b7-4e77-8f12-e2f3c0a22bef-kube-api-access-kjpn2\") pod \"cert-manager-cainjector-559cdd8c87-z8225\" (UID: \"e1d379a5-61b7-4e77-8f12-e2f3c0a22bef\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z8225" Sep 08 16:54:58.551096 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:58.551010 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1d379a5-61b7-4e77-8f12-e2f3c0a22bef-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z8225\" (UID: \"e1d379a5-61b7-4e77-8f12-e2f3c0a22bef\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z8225" Sep 08 16:54:58.560177 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:58.560148 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjpn2\" (UniqueName: \"kubernetes.io/projected/e1d379a5-61b7-4e77-8f12-e2f3c0a22bef-kube-api-access-kjpn2\") pod \"cert-manager-cainjector-559cdd8c87-z8225\" (UID: \"e1d379a5-61b7-4e77-8f12-e2f3c0a22bef\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z8225" Sep 08 16:54:58.561066 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:58.561045 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1d379a5-61b7-4e77-8f12-e2f3c0a22bef-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z8225\" (UID: \"e1d379a5-61b7-4e77-8f12-e2f3c0a22bef\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z8225" Sep 08 16:54:58.687822 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:58.687802 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z8225" Sep 08 16:54:58.972298 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:58.972226 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z8225"] Sep 08 16:54:58.975032 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:54:58.974997 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1d379a5_61b7_4e77_8f12_e2f3c0a22bef.slice/crio-db235f79b5590460a4e2055f5bd19d225ae4edcb78bcb731c9cbbb0f0e20e230 WatchSource:0}: Error finding container db235f79b5590460a4e2055f5bd19d225ae4edcb78bcb731c9cbbb0f0e20e230: Status 404 returned error can't find the container with id db235f79b5590460a4e2055f5bd19d225ae4edcb78bcb731c9cbbb0f0e20e230 Sep 08 16:54:59.911849 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:54:59.911807 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z8225" event={"ID":"e1d379a5-61b7-4e77-8f12-e2f3c0a22bef","Type":"ContainerStarted","Data":"db235f79b5590460a4e2055f5bd19d225ae4edcb78bcb731c9cbbb0f0e20e230"} Sep 08 16:55:01.371745 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:01.371700 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:01.925078 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:01.925039 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z8225" event={"ID":"e1d379a5-61b7-4e77-8f12-e2f3c0a22bef","Type":"ContainerStarted","Data":"9212c3a01ef7859c8d464634b59af2744878d13587f37628e7f010449dac2559"} Sep 08 16:55:01.941760 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:01.941716 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z8225" podStartSLOduration=1.439160984 podStartE2EDuration="3.941702239s" totalImagesPullingTime="2.502541255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:55:01.94050791 +0000 UTC m=+264.677339025" watchObservedRunningTime="2026-09-08 16:55:01.941702239 +0000 UTC m=+264.678533355" Sep 08 16:55:02.234338 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:02.234257 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:02.340570 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:02.340526 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:03.649612 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:03.649575 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrmr9"] Sep 08 16:55:03.652922 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:03.652906 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrmr9" Sep 08 16:55:03.656352 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:03.656330 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:55:03.656352 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:03.656346 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 08 16:55:03.657052 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:03.657035 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-79k29\"" Sep 08 16:55:03.667356 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:03.667332 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrmr9"] Sep 08 16:55:03.689502 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:03.689455 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm2c9\" (UniqueName: \"kubernetes.io/projected/1838d52a-5cb2-4d8d-a9e3-c1d93f64d056-kube-api-access-dm2c9\") pod \"openshift-lws-operator-fd8ccff4c-hrmr9\" (UID: \"1838d52a-5cb2-4d8d-a9e3-c1d93f64d056\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrmr9" Sep 08 16:55:03.689635 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:03.689557 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1838d52a-5cb2-4d8d-a9e3-c1d93f64d056-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hrmr9\" (UID: \"1838d52a-5cb2-4d8d-a9e3-c1d93f64d056\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrmr9" Sep 08 16:55:03.790913 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:03.790867 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1838d52a-5cb2-4d8d-a9e3-c1d93f64d056-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hrmr9\" (UID: \"1838d52a-5cb2-4d8d-a9e3-c1d93f64d056\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrmr9" Sep 08 16:55:03.791100 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:03.790955 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm2c9\" (UniqueName: \"kubernetes.io/projected/1838d52a-5cb2-4d8d-a9e3-c1d93f64d056-kube-api-access-dm2c9\") pod \"openshift-lws-operator-fd8ccff4c-hrmr9\" (UID: \"1838d52a-5cb2-4d8d-a9e3-c1d93f64d056\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrmr9" Sep 08 16:55:03.791346 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:03.791326 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1838d52a-5cb2-4d8d-a9e3-c1d93f64d056-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hrmr9\" (UID: \"1838d52a-5cb2-4d8d-a9e3-c1d93f64d056\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrmr9" Sep 08 16:55:03.799857 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:03.799835 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm2c9\" (UniqueName: \"kubernetes.io/projected/1838d52a-5cb2-4d8d-a9e3-c1d93f64d056-kube-api-access-dm2c9\") pod \"openshift-lws-operator-fd8ccff4c-hrmr9\" (UID: \"1838d52a-5cb2-4d8d-a9e3-c1d93f64d056\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrmr9" Sep 08 16:55:03.961821 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:03.961782 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrmr9" Sep 08 16:55:04.234960 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:04.234936 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrmr9"] Sep 08 16:55:04.237301 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:55:04.237269 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1838d52a_5cb2_4d8d_a9e3_c1d93f64d056.slice/crio-fcf643b3ab9d864a2f760c3c39dfda1b9baa23b7fed134099aef4c5ce8489a2b WatchSource:0}: Error finding container fcf643b3ab9d864a2f760c3c39dfda1b9baa23b7fed134099aef4c5ce8489a2b: Status 404 returned error can't find the container with id fcf643b3ab9d864a2f760c3c39dfda1b9baa23b7fed134099aef4c5ce8489a2b Sep 08 16:55:04.938913 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:04.938875 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrmr9" event={"ID":"1838d52a-5cb2-4d8d-a9e3-c1d93f64d056","Type":"ContainerStarted","Data":"fcf643b3ab9d864a2f760c3c39dfda1b9baa23b7fed134099aef4c5ce8489a2b"} Sep 08 16:55:06.697199 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:06.697166 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:06.949999 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:06.949912 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrmr9" event={"ID":"1838d52a-5cb2-4d8d-a9e3-c1d93f64d056","Type":"ContainerStarted","Data":"5f638dc4f699a8fb02ac196fb51484deef0c364dd80d3a6ac9ef9710a4827151"} Sep 08 16:55:06.975372 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:06.975317 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrmr9" podStartSLOduration=2.091932452 podStartE2EDuration="3.975299802s" totalImagesPullingTime="1.88336735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:55:06.972484252 +0000 UTC m=+269.709315368" watchObservedRunningTime="2026-09-08 16:55:06.975299802 +0000 UTC m=+269.712130917" Sep 08 16:55:07.712947 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:07.712897 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:07.810564 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:07.810529 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:09.146049 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:09.146012 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf"] Sep 08 16:55:09.149520 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:09.149502 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf" Sep 08 16:55:09.151858 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:09.151831 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 16:55:09.151858 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:09.151831 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 16:55:09.152584 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:09.152566 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-v8k2r\"" Sep 08 16:55:09.160063 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:09.160042 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf"] Sep 08 16:55:09.236800 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:09.236765 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlllt\" (UniqueName: \"kubernetes.io/projected/319098d3-04ba-4868-8865-86e6b42b423d-kube-api-access-zlllt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf\" (UID: \"319098d3-04ba-4868-8865-86e6b42b423d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf" Sep 08 16:55:09.236800 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:09.236804 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/319098d3-04ba-4868-8865-86e6b42b423d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf\" (UID: \"319098d3-04ba-4868-8865-86e6b42b423d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf" Sep 08 16:55:09.236996 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:09.236864 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/319098d3-04ba-4868-8865-86e6b42b423d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf\" (UID: \"319098d3-04ba-4868-8865-86e6b42b423d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf" Sep 08 16:55:09.337549 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:09.337514 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/319098d3-04ba-4868-8865-86e6b42b423d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf\" (UID: \"319098d3-04ba-4868-8865-86e6b42b423d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf" Sep 08 16:55:09.337711 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:09.337572 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlllt\" (UniqueName: \"kubernetes.io/projected/319098d3-04ba-4868-8865-86e6b42b423d-kube-api-access-zlllt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf\" (UID: \"319098d3-04ba-4868-8865-86e6b42b423d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf" Sep 08 16:55:09.337711 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:09.337604 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/319098d3-04ba-4868-8865-86e6b42b423d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf\" (UID: \"319098d3-04ba-4868-8865-86e6b42b423d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf" Sep 08 16:55:09.337905 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:09.337884 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/319098d3-04ba-4868-8865-86e6b42b423d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf\" (UID: \"319098d3-04ba-4868-8865-86e6b42b423d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf" Sep 08 16:55:09.337982 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:09.337951 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/319098d3-04ba-4868-8865-86e6b42b423d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf\" (UID: \"319098d3-04ba-4868-8865-86e6b42b423d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf" Sep 08 16:55:09.346448 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:09.346404 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlllt\" (UniqueName: \"kubernetes.io/projected/319098d3-04ba-4868-8865-86e6b42b423d-kube-api-access-zlllt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf\" (UID: \"319098d3-04ba-4868-8865-86e6b42b423d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf" Sep 08 16:55:09.458209 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:09.458173 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf" Sep 08 16:55:09.741458 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:09.741419 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf"] Sep 08 16:55:09.743251 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:55:09.743224 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319098d3_04ba_4868_8865_86e6b42b423d.slice/crio-27231042132ac05765db88cc26577e74e9e3c765090aee7ada74b22ca31aa50f WatchSource:0}: Error finding container 27231042132ac05765db88cc26577e74e9e3c765090aee7ada74b22ca31aa50f: Status 404 returned error can't find the container with id 27231042132ac05765db88cc26577e74e9e3c765090aee7ada74b22ca31aa50f Sep 08 16:55:09.967252 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:09.967216 2565 generic.go:356] "Generic (PLEG): container finished" podID="319098d3-04ba-4868-8865-86e6b42b423d" containerID="c74527bb188ab3760a4ba1b16173a49fffca01f4c703ba0618a239bc838fc23f" exitCode=0 Sep 08 16:55:09.967413 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:09.967285 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf" event={"ID":"319098d3-04ba-4868-8865-86e6b42b423d","Type":"ContainerDied","Data":"c74527bb188ab3760a4ba1b16173a49fffca01f4c703ba0618a239bc838fc23f"} Sep 08 16:55:09.967413 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:09.967321 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf" event={"ID":"319098d3-04ba-4868-8865-86e6b42b423d","Type":"ContainerStarted","Data":"27231042132ac05765db88cc26577e74e9e3c765090aee7ada74b22ca31aa50f"} Sep 08 16:55:09.988017 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:09.987983 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:10.974001 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:10.973969 2565 generic.go:356] "Generic (PLEG): container finished" podID="319098d3-04ba-4868-8865-86e6b42b423d" containerID="f8fba00d2441b410b326c3be1ac4dbd20f268324508a523ac75b2a1a85961803" exitCode=0 Sep 08 16:55:10.974341 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:10.974033 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf" event={"ID":"319098d3-04ba-4868-8865-86e6b42b423d","Type":"ContainerDied","Data":"f8fba00d2441b410b326c3be1ac4dbd20f268324508a523ac75b2a1a85961803"} Sep 08 16:55:11.981149 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:11.981108 2565 generic.go:356] "Generic (PLEG): container finished" podID="319098d3-04ba-4868-8865-86e6b42b423d" containerID="9d1cdd21129c293789f1d089a04eb03302bdc120b1b112b0424422594be33ccb" exitCode=0 Sep 08 16:55:11.981559 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:11.981171 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf" event={"ID":"319098d3-04ba-4868-8865-86e6b42b423d","Type":"ContainerDied","Data":"9d1cdd21129c293789f1d089a04eb03302bdc120b1b112b0424422594be33ccb"} Sep 08 16:55:13.131889 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:13.131853 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf" Sep 08 16:55:13.168069 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:13.168039 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/319098d3-04ba-4868-8865-86e6b42b423d-util\") pod \"319098d3-04ba-4868-8865-86e6b42b423d\" (UID: \"319098d3-04ba-4868-8865-86e6b42b423d\") " Sep 08 16:55:13.168234 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:13.168078 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/319098d3-04ba-4868-8865-86e6b42b423d-bundle\") pod \"319098d3-04ba-4868-8865-86e6b42b423d\" (UID: \"319098d3-04ba-4868-8865-86e6b42b423d\") " Sep 08 16:55:13.168234 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:13.168148 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlllt\" (UniqueName: \"kubernetes.io/projected/319098d3-04ba-4868-8865-86e6b42b423d-kube-api-access-zlllt\") pod \"319098d3-04ba-4868-8865-86e6b42b423d\" (UID: \"319098d3-04ba-4868-8865-86e6b42b423d\") " Sep 08 16:55:13.168816 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:13.168785 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319098d3-04ba-4868-8865-86e6b42b423d-bundle" (OuterVolumeSpecName: "bundle") pod "319098d3-04ba-4868-8865-86e6b42b423d" (UID: "319098d3-04ba-4868-8865-86e6b42b423d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:55:13.170322 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:13.170294 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319098d3-04ba-4868-8865-86e6b42b423d-kube-api-access-zlllt" (OuterVolumeSpecName: "kube-api-access-zlllt") pod "319098d3-04ba-4868-8865-86e6b42b423d" (UID: "319098d3-04ba-4868-8865-86e6b42b423d"). InnerVolumeSpecName "kube-api-access-zlllt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:55:13.177276 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:13.177244 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319098d3-04ba-4868-8865-86e6b42b423d-util" (OuterVolumeSpecName: "util") pod "319098d3-04ba-4868-8865-86e6b42b423d" (UID: "319098d3-04ba-4868-8865-86e6b42b423d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:55:13.268980 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:13.268914 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlllt\" (UniqueName: \"kubernetes.io/projected/319098d3-04ba-4868-8865-86e6b42b423d-kube-api-access-zlllt\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:55:13.268980 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:13.268940 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/319098d3-04ba-4868-8865-86e6b42b423d-util\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:55:13.268980 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:13.268970 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/319098d3-04ba-4868-8865-86e6b42b423d-bundle\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:55:13.992303 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:13.992265 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf" event={"ID":"319098d3-04ba-4868-8865-86e6b42b423d","Type":"ContainerDied","Data":"27231042132ac05765db88cc26577e74e9e3c765090aee7ada74b22ca31aa50f"} Sep 08 16:55:13.992492 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:13.992325 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27231042132ac05765db88cc26577e74e9e3c765090aee7ada74b22ca31aa50f" Sep 08 16:55:13.992492 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:13.992278 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg7tnf" Sep 08 16:55:22.518631 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:22.518597 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:22.667697 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:22.667656 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:23.257130 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.257095 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:23.492246 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.492208 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd"] Sep 08 16:55:23.492587 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.492569 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="319098d3-04ba-4868-8865-86e6b42b423d" containerName="extract" Sep 08 16:55:23.492686 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.492589 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="319098d3-04ba-4868-8865-86e6b42b423d" containerName="extract" Sep 08 16:55:23.492686 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.492607 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="319098d3-04ba-4868-8865-86e6b42b423d" containerName="util" Sep 08 16:55:23.492686 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.492616 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="319098d3-04ba-4868-8865-86e6b42b423d" containerName="util" Sep 08 16:55:23.492686 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.492631 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="319098d3-04ba-4868-8865-86e6b42b423d" containerName="pull" Sep 08 16:55:23.492686 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.492639 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="319098d3-04ba-4868-8865-86e6b42b423d" containerName="pull" Sep 08 16:55:23.492926 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.492738 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="319098d3-04ba-4868-8865-86e6b42b423d" containerName="extract" Sep 08 16:55:23.498886 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.498863 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd" Sep 08 16:55:23.504497 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.504472 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-v8k2r\"" Sep 08 16:55:23.504497 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.504492 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 16:55:23.505356 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.505229 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 16:55:23.523190 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.523131 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd"] Sep 08 16:55:23.549097 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.549065 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7m6j\" (UniqueName: \"kubernetes.io/projected/88a8f753-63f7-46ac-85fc-9fdee2db5589-kube-api-access-j7m6j\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd\" (UID: \"88a8f753-63f7-46ac-85fc-9fdee2db5589\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd" Sep 08 16:55:23.549225 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.549118 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88a8f753-63f7-46ac-85fc-9fdee2db5589-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd\" (UID: \"88a8f753-63f7-46ac-85fc-9fdee2db5589\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd" Sep 08 16:55:23.549225 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.549181 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88a8f753-63f7-46ac-85fc-9fdee2db5589-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd\" (UID: \"88a8f753-63f7-46ac-85fc-9fdee2db5589\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd" Sep 08 16:55:23.650110 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.650070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7m6j\" (UniqueName: \"kubernetes.io/projected/88a8f753-63f7-46ac-85fc-9fdee2db5589-kube-api-access-j7m6j\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd\" (UID: \"88a8f753-63f7-46ac-85fc-9fdee2db5589\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd" Sep 08 16:55:23.650325 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.650128 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88a8f753-63f7-46ac-85fc-9fdee2db5589-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd\" (UID: \"88a8f753-63f7-46ac-85fc-9fdee2db5589\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd" Sep 08 16:55:23.650325 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.650156 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88a8f753-63f7-46ac-85fc-9fdee2db5589-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd\" (UID: \"88a8f753-63f7-46ac-85fc-9fdee2db5589\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd" Sep 08 16:55:23.650537 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.650519 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88a8f753-63f7-46ac-85fc-9fdee2db5589-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd\" (UID: \"88a8f753-63f7-46ac-85fc-9fdee2db5589\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd" Sep 08 16:55:23.650580 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.650549 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88a8f753-63f7-46ac-85fc-9fdee2db5589-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd\" (UID: \"88a8f753-63f7-46ac-85fc-9fdee2db5589\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd" Sep 08 16:55:23.664505 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.664480 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7m6j\" (UniqueName: \"kubernetes.io/projected/88a8f753-63f7-46ac-85fc-9fdee2db5589-kube-api-access-j7m6j\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd\" (UID: \"88a8f753-63f7-46ac-85fc-9fdee2db5589\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd" Sep 08 16:55:23.808039 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.807943 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd" Sep 08 16:55:23.995289 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:23.995247 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:24.110827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.110791 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd"] Sep 08 16:55:24.113057 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:55:24.113029 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a8f753_63f7_46ac_85fc_9fdee2db5589.slice/crio-b2e261a2101d3333e2b5c35b5242d773dab259a4e0b7932f79e288d8914bee0e WatchSource:0}: Error finding container b2e261a2101d3333e2b5c35b5242d773dab259a4e0b7932f79e288d8914bee0e: Status 404 returned error can't find the container with id b2e261a2101d3333e2b5c35b5242d773dab259a4e0b7932f79e288d8914bee0e Sep 08 16:55:24.167672 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.167639 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:24.217737 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.217709 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72"] Sep 08 16:55:24.221385 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.221364 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" Sep 08 16:55:24.226457 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.226418 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 08 16:55:24.226574 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.226434 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 08 16:55:24.226718 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.226702 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 08 16:55:24.227346 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.227329 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 08 16:55:24.233112 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.233093 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-l98dr\"" Sep 08 16:55:24.248412 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.248382 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72"] Sep 08 16:55:24.258524 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.258498 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4490dbae-ceb9-4f64-a491-fc2f20ecc243-manifests\") pod \"opendatahub-operator-controller-manager-84f58c54b-62m72\" (UID: \"4490dbae-ceb9-4f64-a491-fc2f20ecc243\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" Sep 08 16:55:24.258642 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.258533 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4490dbae-ceb9-4f64-a491-fc2f20ecc243-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84f58c54b-62m72\" (UID: \"4490dbae-ceb9-4f64-a491-fc2f20ecc243\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" Sep 08 16:55:24.258642 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.258553 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4490dbae-ceb9-4f64-a491-fc2f20ecc243-tmp\") pod \"opendatahub-operator-controller-manager-84f58c54b-62m72\" (UID: \"4490dbae-ceb9-4f64-a491-fc2f20ecc243\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" Sep 08 16:55:24.258642 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.258597 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4490dbae-ceb9-4f64-a491-fc2f20ecc243-webhook-cert\") pod \"opendatahub-operator-controller-manager-84f58c54b-62m72\" (UID: \"4490dbae-ceb9-4f64-a491-fc2f20ecc243\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" Sep 08 16:55:24.258642 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.258620 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpbz2\" (UniqueName: \"kubernetes.io/projected/4490dbae-ceb9-4f64-a491-fc2f20ecc243-kube-api-access-dpbz2\") pod \"opendatahub-operator-controller-manager-84f58c54b-62m72\" (UID: \"4490dbae-ceb9-4f64-a491-fc2f20ecc243\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" Sep 08 16:55:24.359196 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.359157 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4490dbae-ceb9-4f64-a491-fc2f20ecc243-manifests\") pod \"opendatahub-operator-controller-manager-84f58c54b-62m72\" (UID: \"4490dbae-ceb9-4f64-a491-fc2f20ecc243\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" Sep 08 16:55:24.359196 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.359200 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4490dbae-ceb9-4f64-a491-fc2f20ecc243-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84f58c54b-62m72\" (UID: \"4490dbae-ceb9-4f64-a491-fc2f20ecc243\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" Sep 08 16:55:24.359491 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.359220 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4490dbae-ceb9-4f64-a491-fc2f20ecc243-tmp\") pod \"opendatahub-operator-controller-manager-84f58c54b-62m72\" (UID: \"4490dbae-ceb9-4f64-a491-fc2f20ecc243\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" Sep 08 16:55:24.359491 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.359241 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4490dbae-ceb9-4f64-a491-fc2f20ecc243-webhook-cert\") pod \"opendatahub-operator-controller-manager-84f58c54b-62m72\" (UID: \"4490dbae-ceb9-4f64-a491-fc2f20ecc243\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" Sep 08 16:55:24.359491 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.359259 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpbz2\" (UniqueName: \"kubernetes.io/projected/4490dbae-ceb9-4f64-a491-fc2f20ecc243-kube-api-access-dpbz2\") pod \"opendatahub-operator-controller-manager-84f58c54b-62m72\" (UID: \"4490dbae-ceb9-4f64-a491-fc2f20ecc243\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" Sep 08 16:55:24.359745 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.359718 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4490dbae-ceb9-4f64-a491-fc2f20ecc243-manifests\") pod \"opendatahub-operator-controller-manager-84f58c54b-62m72\" (UID: \"4490dbae-ceb9-4f64-a491-fc2f20ecc243\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" Sep 08 16:55:24.361544 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.361481 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4490dbae-ceb9-4f64-a491-fc2f20ecc243-tmp\") pod \"opendatahub-operator-controller-manager-84f58c54b-62m72\" (UID: \"4490dbae-ceb9-4f64-a491-fc2f20ecc243\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" Sep 08 16:55:24.361787 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.361767 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4490dbae-ceb9-4f64-a491-fc2f20ecc243-webhook-cert\") pod \"opendatahub-operator-controller-manager-84f58c54b-62m72\" (UID: \"4490dbae-ceb9-4f64-a491-fc2f20ecc243\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" Sep 08 16:55:24.361849 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.361784 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4490dbae-ceb9-4f64-a491-fc2f20ecc243-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84f58c54b-62m72\" (UID: \"4490dbae-ceb9-4f64-a491-fc2f20ecc243\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" Sep 08 16:55:24.375651 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.375620 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpbz2\" (UniqueName: \"kubernetes.io/projected/4490dbae-ceb9-4f64-a491-fc2f20ecc243-kube-api-access-dpbz2\") pod \"opendatahub-operator-controller-manager-84f58c54b-62m72\" (UID: \"4490dbae-ceb9-4f64-a491-fc2f20ecc243\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" Sep 08 16:55:24.568051 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.568008 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" Sep 08 16:55:24.658688 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.658654 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:24.865738 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.865710 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72"] Sep 08 16:55:24.868182 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:55:24.868153 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4490dbae_ceb9_4f64_a491_fc2f20ecc243.slice/crio-b6113567b9ed6f6a7e9539f732e7b6e1dbe76ce66b4370ffe749a3c6f827f4ff WatchSource:0}: Error finding container b6113567b9ed6f6a7e9539f732e7b6e1dbe76ce66b4370ffe749a3c6f827f4ff: Status 404 returned error can't find the container with id b6113567b9ed6f6a7e9539f732e7b6e1dbe76ce66b4370ffe749a3c6f827f4ff Sep 08 16:55:24.947586 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:24.947551 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:25.041501 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:25.041407 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" event={"ID":"4490dbae-ceb9-4f64-a491-fc2f20ecc243","Type":"ContainerStarted","Data":"b6113567b9ed6f6a7e9539f732e7b6e1dbe76ce66b4370ffe749a3c6f827f4ff"} Sep 08 16:55:25.042910 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:25.042885 2565 generic.go:356] "Generic (PLEG): container finished" podID="88a8f753-63f7-46ac-85fc-9fdee2db5589" containerID="e38fb271f16cd2d5031d6a0dd81bd99c16b17da7102e4651c129b464c647127e" exitCode=0 Sep 08 16:55:25.043024 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:25.042948 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd" event={"ID":"88a8f753-63f7-46ac-85fc-9fdee2db5589","Type":"ContainerDied","Data":"e38fb271f16cd2d5031d6a0dd81bd99c16b17da7102e4651c129b464c647127e"} Sep 08 16:55:25.043024 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:25.042980 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd" event={"ID":"88a8f753-63f7-46ac-85fc-9fdee2db5589","Type":"ContainerStarted","Data":"b2e261a2101d3333e2b5c35b5242d773dab259a4e0b7932f79e288d8914bee0e"} Sep 08 16:55:25.139389 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:25.139293 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:25.313218 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:25.313177 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:25.434124 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:25.434088 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:25.581517 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:25.581475 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:27.054518 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:27.054481 2565 generic.go:356] "Generic (PLEG): container finished" podID="88a8f753-63f7-46ac-85fc-9fdee2db5589" containerID="10a3f1adb6d1fad3b9d85e122b66b8a68f2fe2b14cc9662312d3526450cd6d7f" exitCode=0 Sep 08 16:55:27.054960 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:27.054563 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd" event={"ID":"88a8f753-63f7-46ac-85fc-9fdee2db5589","Type":"ContainerDied","Data":"10a3f1adb6d1fad3b9d85e122b66b8a68f2fe2b14cc9662312d3526450cd6d7f"} Sep 08 16:55:28.067642 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:28.067605 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd" event={"ID":"88a8f753-63f7-46ac-85fc-9fdee2db5589","Type":"ContainerStarted","Data":"d86f10804cec41f810c4067428ba6f7b6ad9cf57f38ad8128d53fc57bd0b6ff4"} Sep 08 16:55:28.122506 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:28.122418 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd" podStartSLOduration=4.18726644 podStartE2EDuration="5.122394458s" totalImagesPullingTime="935.128018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:55:28.121560224 +0000 UTC m=+290.858391339" watchObservedRunningTime="2026-09-08 16:55:28.122394458 +0000 UTC m=+290.859225574" Sep 08 16:55:28.472017 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:28.471981 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:28.803915 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:28.803824 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:29.074123 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:29.074035 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" event={"ID":"4490dbae-ceb9-4f64-a491-fc2f20ecc243","Type":"ContainerStarted","Data":"0776e296e87a9d9988793d181121505cc469cc7b26ffae86456870117bd79ab0"} Sep 08 16:55:29.076334 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:29.076300 2565 generic.go:356] "Generic (PLEG): container finished" podID="88a8f753-63f7-46ac-85fc-9fdee2db5589" containerID="d86f10804cec41f810c4067428ba6f7b6ad9cf57f38ad8128d53fc57bd0b6ff4" exitCode=0 Sep 08 16:55:29.076470 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:29.076372 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd" event={"ID":"88a8f753-63f7-46ac-85fc-9fdee2db5589","Type":"ContainerDied","Data":"d86f10804cec41f810c4067428ba6f7b6ad9cf57f38ad8128d53fc57bd0b6ff4"} Sep 08 16:55:30.083072 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:30.083034 2565 generic.go:356] "Generic (PLEG): container finished" podID="4490dbae-ceb9-4f64-a491-fc2f20ecc243" containerID="0776e296e87a9d9988793d181121505cc469cc7b26ffae86456870117bd79ab0" exitCode=0 Sep 08 16:55:30.083571 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:30.083126 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" event={"ID":"4490dbae-ceb9-4f64-a491-fc2f20ecc243","Type":"ContainerDied","Data":"0776e296e87a9d9988793d181121505cc469cc7b26ffae86456870117bd79ab0"} Sep 08 16:55:30.206357 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:30.206333 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd" Sep 08 16:55:30.308759 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:30.308723 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7m6j\" (UniqueName: \"kubernetes.io/projected/88a8f753-63f7-46ac-85fc-9fdee2db5589-kube-api-access-j7m6j\") pod \"88a8f753-63f7-46ac-85fc-9fdee2db5589\" (UID: \"88a8f753-63f7-46ac-85fc-9fdee2db5589\") " Sep 08 16:55:30.308759 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:30.308766 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88a8f753-63f7-46ac-85fc-9fdee2db5589-bundle\") pod \"88a8f753-63f7-46ac-85fc-9fdee2db5589\" (UID: \"88a8f753-63f7-46ac-85fc-9fdee2db5589\") " Sep 08 16:55:30.309001 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:30.308904 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88a8f753-63f7-46ac-85fc-9fdee2db5589-util\") pod \"88a8f753-63f7-46ac-85fc-9fdee2db5589\" (UID: \"88a8f753-63f7-46ac-85fc-9fdee2db5589\") " Sep 08 16:55:30.309516 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:30.309493 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a8f753-63f7-46ac-85fc-9fdee2db5589-bundle" (OuterVolumeSpecName: "bundle") pod "88a8f753-63f7-46ac-85fc-9fdee2db5589" (UID: "88a8f753-63f7-46ac-85fc-9fdee2db5589"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:55:30.310998 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:30.310977 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a8f753-63f7-46ac-85fc-9fdee2db5589-kube-api-access-j7m6j" (OuterVolumeSpecName: "kube-api-access-j7m6j") pod "88a8f753-63f7-46ac-85fc-9fdee2db5589" (UID: "88a8f753-63f7-46ac-85fc-9fdee2db5589"). InnerVolumeSpecName "kube-api-access-j7m6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:55:30.313478 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:30.313433 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a8f753-63f7-46ac-85fc-9fdee2db5589-util" (OuterVolumeSpecName: "util") pod "88a8f753-63f7-46ac-85fc-9fdee2db5589" (UID: "88a8f753-63f7-46ac-85fc-9fdee2db5589"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:55:30.409569 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:30.409474 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88a8f753-63f7-46ac-85fc-9fdee2db5589-util\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:55:30.409569 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:30.409519 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7m6j\" (UniqueName: \"kubernetes.io/projected/88a8f753-63f7-46ac-85fc-9fdee2db5589-kube-api-access-j7m6j\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:55:30.409569 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:30.409530 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88a8f753-63f7-46ac-85fc-9fdee2db5589-bundle\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:55:31.090648 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:31.090613 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" event={"ID":"4490dbae-ceb9-4f64-a491-fc2f20ecc243","Type":"ContainerStarted","Data":"d64446b7ca7c778447423658e97cc22a3de1ea71121ed520cefedec234d43b4f"} Sep 08 16:55:31.091101 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:31.090765 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" Sep 08 16:55:31.092780 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:31.092761 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd" Sep 08 16:55:31.092891 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:31.092759 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0z5fcd" event={"ID":"88a8f753-63f7-46ac-85fc-9fdee2db5589","Type":"ContainerDied","Data":"b2e261a2101d3333e2b5c35b5242d773dab259a4e0b7932f79e288d8914bee0e"} Sep 08 16:55:31.092891 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:31.092879 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2e261a2101d3333e2b5c35b5242d773dab259a4e0b7932f79e288d8914bee0e" Sep 08 16:55:31.131444 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:31.131372 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" podStartSLOduration=1.956193426 podStartE2EDuration="7.13135712s" totalImagesPullingTime="4.028546717s" totalInitContainerRuntime="1.146616977s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:55:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:55:31.130163083 +0000 UTC m=+293.866994199" watchObservedRunningTime="2026-09-08 16:55:31.13135712 +0000 UTC m=+293.868188236" Sep 08 16:55:34.637709 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:34.637676 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:34.779370 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:34.779327 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:35.724888 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:35.724854 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:35.865481 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:35.865448 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:37.691450 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:37.691400 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/2.log" Sep 08 16:55:37.697482 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:37.697403 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/2.log" Sep 08 16:55:37.700878 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:37.700858 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xxnl_bf53ac63-f864-402b-bf12-45b289e7e13d/ovn-acl-logging/0.log" Sep 08 16:55:37.704593 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:37.704574 2565 kubelet.go:1648] "Image garbage collection succeeded" Sep 08 16:55:37.706717 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:37.706699 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xxnl_bf53ac63-f864-402b-bf12-45b289e7e13d/ovn-acl-logging/0.log" Sep 08 16:55:42.099794 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:42.099762 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-62m72" Sep 08 16:55:44.648752 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:44.648719 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:44.863914 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:44.863882 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:45.523486 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.523442 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:45.545881 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.545848 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d8677cc5f-pcp9g"] Sep 08 16:55:45.546192 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.546179 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a8f753-63f7-46ac-85fc-9fdee2db5589" containerName="pull" Sep 08 16:55:45.546234 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.546194 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a8f753-63f7-46ac-85fc-9fdee2db5589" containerName="pull" Sep 08 16:55:45.546234 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.546205 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a8f753-63f7-46ac-85fc-9fdee2db5589" containerName="util" Sep 08 16:55:45.546234 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.546211 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a8f753-63f7-46ac-85fc-9fdee2db5589" containerName="util" Sep 08 16:55:45.546234 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.546223 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a8f753-63f7-46ac-85fc-9fdee2db5589" containerName="extract" Sep 08 16:55:45.546234 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.546230 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a8f753-63f7-46ac-85fc-9fdee2db5589" containerName="extract" Sep 08 16:55:45.546390 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.546288 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="88a8f753-63f7-46ac-85fc-9fdee2db5589" containerName="extract" Sep 08 16:55:45.549540 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.549524 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-pcp9g" Sep 08 16:55:45.560687 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.560665 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 08 16:55:45.561011 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.560995 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 08 16:55:45.626265 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.626234 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d8677cc5f-pcp9g"] Sep 08 16:55:45.627982 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.627957 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j66mw\" (UniqueName: \"kubernetes.io/projected/7e359e34-b048-4110-9fc0-149b12c3363a-kube-api-access-j66mw\") pod \"kube-auth-proxy-7d8677cc5f-pcp9g\" (UID: \"7e359e34-b048-4110-9fc0-149b12c3363a\") " pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-pcp9g" Sep 08 16:55:45.628079 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.628008 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e359e34-b048-4110-9fc0-149b12c3363a-tls-certs\") pod \"kube-auth-proxy-7d8677cc5f-pcp9g\" (UID: \"7e359e34-b048-4110-9fc0-149b12c3363a\") " pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-pcp9g" Sep 08 16:55:45.628148 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.628129 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e359e34-b048-4110-9fc0-149b12c3363a-tmp\") pod \"kube-auth-proxy-7d8677cc5f-pcp9g\" (UID: \"7e359e34-b048-4110-9fc0-149b12c3363a\") " pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-pcp9g" Sep 08 16:55:45.728561 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.728525 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e359e34-b048-4110-9fc0-149b12c3363a-tls-certs\") pod \"kube-auth-proxy-7d8677cc5f-pcp9g\" (UID: \"7e359e34-b048-4110-9fc0-149b12c3363a\") " pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-pcp9g" Sep 08 16:55:45.728929 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.728596 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e359e34-b048-4110-9fc0-149b12c3363a-tmp\") pod \"kube-auth-proxy-7d8677cc5f-pcp9g\" (UID: \"7e359e34-b048-4110-9fc0-149b12c3363a\") " pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-pcp9g" Sep 08 16:55:45.728929 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.728632 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j66mw\" (UniqueName: \"kubernetes.io/projected/7e359e34-b048-4110-9fc0-149b12c3363a-kube-api-access-j66mw\") pod \"kube-auth-proxy-7d8677cc5f-pcp9g\" (UID: \"7e359e34-b048-4110-9fc0-149b12c3363a\") " pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-pcp9g" Sep 08 16:55:45.730806 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.730776 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e359e34-b048-4110-9fc0-149b12c3363a-tmp\") pod \"kube-auth-proxy-7d8677cc5f-pcp9g\" (UID: \"7e359e34-b048-4110-9fc0-149b12c3363a\") " pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-pcp9g" Sep 08 16:55:45.731116 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.731094 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e359e34-b048-4110-9fc0-149b12c3363a-tls-certs\") pod \"kube-auth-proxy-7d8677cc5f-pcp9g\" (UID: \"7e359e34-b048-4110-9fc0-149b12c3363a\") " pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-pcp9g" Sep 08 16:55:45.754469 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.754415 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j66mw\" (UniqueName: \"kubernetes.io/projected/7e359e34-b048-4110-9fc0-149b12c3363a-kube-api-access-j66mw\") pod \"kube-auth-proxy-7d8677cc5f-pcp9g\" (UID: \"7e359e34-b048-4110-9fc0-149b12c3363a\") " pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-pcp9g" Sep 08 16:55:45.858561 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:45.858483 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-pcp9g" Sep 08 16:55:46.095947 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:46.095899 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:46.190495 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:46.190466 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d8677cc5f-pcp9g"] Sep 08 16:55:46.192049 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:55:46.192019 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e359e34_b048_4110_9fc0_149b12c3363a.slice/crio-07182ff2bbbd48e42191e58b0fefeb3d89e947306a19a2132f4891d79cd0e024 WatchSource:0}: Error finding container 07182ff2bbbd48e42191e58b0fefeb3d89e947306a19a2132f4891d79cd0e024: Status 404 returned error can't find the container with id 07182ff2bbbd48e42191e58b0fefeb3d89e947306a19a2132f4891d79cd0e024 Sep 08 16:55:46.194288 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:46.194084 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 16:55:47.168395 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:47.168348 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-pcp9g" event={"ID":"7e359e34-b048-4110-9fc0-149b12c3363a","Type":"ContainerStarted","Data":"07182ff2bbbd48e42191e58b0fefeb3d89e947306a19a2132f4891d79cd0e024"} Sep 08 16:55:49.179471 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.179436 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-pcp9g" event={"ID":"7e359e34-b048-4110-9fc0-149b12c3363a","Type":"ContainerStarted","Data":"69d33529c2c3fe70f964f48db8542780fb68911825f11c683d6a7ef9b5e1437d"} Sep 08 16:55:49.538294 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.538241 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:49.590769 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.590709 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-pcp9g" podStartSLOduration=1.827404305 podStartE2EDuration="4.590691588s" totalImagesPullingTime="2.763287283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:55:49.215453788 +0000 UTC m=+311.952284905" watchObservedRunningTime="2026-09-08 16:55:49.590691588 +0000 UTC m=+312.327522703" Sep 08 16:55:49.591362 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.591339 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j"] Sep 08 16:55:49.595043 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.595023 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:49.597963 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.597944 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 08 16:55:49.598304 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.598286 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 08 16:55:49.598396 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.598308 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 08 16:55:49.598396 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.598321 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dsdcl\"" Sep 08 16:55:49.599188 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.599172 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 08 16:55:49.618063 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.618033 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j"] Sep 08 16:55:49.655263 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.655224 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:49.767931 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.767899 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-c4z6j\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:49.767931 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.767931 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m9vn\" (UniqueName: \"kubernetes.io/projected/460d1a33-2dec-4313-9bb7-e5fe86e7effa-kube-api-access-7m9vn\") pod \"istiod-openshift-gateway-7b7995d5cd-c4z6j\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:49.768168 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.767956 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/460d1a33-2dec-4313-9bb7-e5fe86e7effa-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-c4z6j\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:49.768168 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.768060 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-c4z6j\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:49.768168 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.768095 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-c4z6j\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:49.768292 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.768188 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-c4z6j\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:49.768292 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.768218 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/460d1a33-2dec-4313-9bb7-e5fe86e7effa-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-c4z6j\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:49.869344 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.869253 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m9vn\" (UniqueName: \"kubernetes.io/projected/460d1a33-2dec-4313-9bb7-e5fe86e7effa-kube-api-access-7m9vn\") pod \"istiod-openshift-gateway-7b7995d5cd-c4z6j\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:49.869344 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.869300 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/460d1a33-2dec-4313-9bb7-e5fe86e7effa-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-c4z6j\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:49.869591 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.869346 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-c4z6j\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:49.869591 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.869373 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-c4z6j\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:49.869591 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.869476 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-c4z6j\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:49.869591 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.869500 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/460d1a33-2dec-4313-9bb7-e5fe86e7effa-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-c4z6j\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:49.869591 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.869570 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-c4z6j\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:49.870215 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.870184 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-c4z6j\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:49.872022 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.871996 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/460d1a33-2dec-4313-9bb7-e5fe86e7effa-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-c4z6j\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:49.872109 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.872045 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/460d1a33-2dec-4313-9bb7-e5fe86e7effa-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-c4z6j\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:49.872152 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.872120 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-c4z6j\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:49.872152 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.872142 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-c4z6j\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:49.880272 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.880247 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-c4z6j\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:49.880366 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.880285 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m9vn\" (UniqueName: \"kubernetes.io/projected/460d1a33-2dec-4313-9bb7-e5fe86e7effa-kube-api-access-7m9vn\") pod \"istiod-openshift-gateway-7b7995d5cd-c4z6j\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:49.904330 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:49.904285 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:50.200875 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:50.200590 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j"] Sep 08 16:55:50.203398 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:55:50.203363 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460d1a33_2dec_4313_9bb7_e5fe86e7effa.slice/crio-54d6409572f34edba8183e146b599e067874daa2f3e8a5b2ca1c3e64ef31c32b WatchSource:0}: Error finding container 54d6409572f34edba8183e146b599e067874daa2f3e8a5b2ca1c3e64ef31c32b: Status 404 returned error can't find the container with id 54d6409572f34edba8183e146b599e067874daa2f3e8a5b2ca1c3e64ef31c32b Sep 08 16:55:51.192314 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:51.192267 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" event={"ID":"460d1a33-2dec-4313-9bb7-e5fe86e7effa","Type":"ContainerStarted","Data":"54d6409572f34edba8183e146b599e067874daa2f3e8a5b2ca1c3e64ef31c32b"} Sep 08 16:55:51.502499 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:51.502312 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:52.784262 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:52.784214 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 16:55:52.784574 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:52.784295 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 16:55:53.204236 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:53.204198 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" event={"ID":"460d1a33-2dec-4313-9bb7-e5fe86e7effa","Type":"ContainerStarted","Data":"10cacfd891971ea224a20f41c2a894744dfd46a26c5596be5e69b64e8bc68562"} Sep 08 16:55:53.204442 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:53.204344 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:53.205962 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:53.205934 2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-c4z6j container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 16:55:53.206056 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:53.205983 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" podUID="460d1a33-2dec-4313-9bb7-e5fe86e7effa" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 16:55:53.246977 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:53.246919 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" podStartSLOduration=1.668785142 podStartE2EDuration="4.246902507s" totalImagesPullingTime="2.578117365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:55:53.245832373 +0000 UTC m=+315.982663620" watchObservedRunningTime="2026-09-08 16:55:53.246902507 +0000 UTC m=+315.983733623" Sep 08 16:55:53.499312 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:53.499220 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:53.789987 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:53.789890 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:54.210311 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:54.210276 2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-c4z6j container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 16:55:54.210507 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:54.210333 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" podUID="460d1a33-2dec-4313-9bb7-e5fe86e7effa" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 16:55:54.764811 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:54.764772 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:55.017146 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:55.017063 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:55.211129 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:55.211076 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:55.723318 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:55.723277 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:56.031943 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:56.031850 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:56.250187 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:56.250156 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:57.048238 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:57.048202 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:55:57.211338 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:57.211308 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:55:57.484778 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:55:57.484743 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:56:08.595984 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.595942 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-s8z86"] Sep 08 16:56:08.599473 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.599450 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s8z86" Sep 08 16:56:08.603919 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.603884 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:56:08.605130 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.605110 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-ctjd9\"" Sep 08 16:56:08.605234 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.605151 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 08 16:56:08.609682 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.609663 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-q788k"] Sep 08 16:56:08.612607 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.612592 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q788k" Sep 08 16:56:08.620044 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.620021 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 08 16:56:08.620268 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.620237 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-lr9db\"" Sep 08 16:56:08.620383 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.620263 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kg9b\" (UniqueName: \"kubernetes.io/projected/517d507a-eadc-4cf7-9193-9b9bdfcefe66-kube-api-access-4kg9b\") pod \"kserve-controller-manager-d6f4b65b8-s8z86\" (UID: \"517d507a-eadc-4cf7-9193-9b9bdfcefe66\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s8z86" Sep 08 16:56:08.620383 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.620306 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6czd\" (UniqueName: \"kubernetes.io/projected/d2bd1bab-bda5-45cf-8812-2a3e6b2026ba-kube-api-access-c6czd\") pod \"llmisvc-controller-manager-777c6bd7d5-q788k\" (UID: \"d2bd1bab-bda5-45cf-8812-2a3e6b2026ba\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q788k" Sep 08 16:56:08.620512 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.620470 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/517d507a-eadc-4cf7-9193-9b9bdfcefe66-cert\") pod \"kserve-controller-manager-d6f4b65b8-s8z86\" (UID: \"517d507a-eadc-4cf7-9193-9b9bdfcefe66\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s8z86" Sep 08 16:56:08.620512 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.620504 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2bd1bab-bda5-45cf-8812-2a3e6b2026ba-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q788k\" (UID: \"d2bd1bab-bda5-45cf-8812-2a3e6b2026ba\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q788k" Sep 08 16:56:08.629730 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.629705 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-s8z86"] Sep 08 16:56:08.635856 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.635835 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-q788k"] Sep 08 16:56:08.720881 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.720848 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6czd\" (UniqueName: \"kubernetes.io/projected/d2bd1bab-bda5-45cf-8812-2a3e6b2026ba-kube-api-access-c6czd\") pod \"llmisvc-controller-manager-777c6bd7d5-q788k\" (UID: \"d2bd1bab-bda5-45cf-8812-2a3e6b2026ba\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q788k" Sep 08 16:56:08.721069 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.720921 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/517d507a-eadc-4cf7-9193-9b9bdfcefe66-cert\") pod \"kserve-controller-manager-d6f4b65b8-s8z86\" (UID: \"517d507a-eadc-4cf7-9193-9b9bdfcefe66\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s8z86" Sep 08 16:56:08.721069 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.720940 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2bd1bab-bda5-45cf-8812-2a3e6b2026ba-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q788k\" (UID: \"d2bd1bab-bda5-45cf-8812-2a3e6b2026ba\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q788k" Sep 08 16:56:08.721069 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.720976 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kg9b\" (UniqueName: \"kubernetes.io/projected/517d507a-eadc-4cf7-9193-9b9bdfcefe66-kube-api-access-4kg9b\") pod \"kserve-controller-manager-d6f4b65b8-s8z86\" (UID: \"517d507a-eadc-4cf7-9193-9b9bdfcefe66\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s8z86" Sep 08 16:56:08.723612 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.723585 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/517d507a-eadc-4cf7-9193-9b9bdfcefe66-cert\") pod \"kserve-controller-manager-d6f4b65b8-s8z86\" (UID: \"517d507a-eadc-4cf7-9193-9b9bdfcefe66\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s8z86" Sep 08 16:56:08.723612 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.723603 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2bd1bab-bda5-45cf-8812-2a3e6b2026ba-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q788k\" (UID: \"d2bd1bab-bda5-45cf-8812-2a3e6b2026ba\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q788k" Sep 08 16:56:08.739874 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.739840 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kg9b\" (UniqueName: \"kubernetes.io/projected/517d507a-eadc-4cf7-9193-9b9bdfcefe66-kube-api-access-4kg9b\") pod \"kserve-controller-manager-d6f4b65b8-s8z86\" (UID: \"517d507a-eadc-4cf7-9193-9b9bdfcefe66\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s8z86" Sep 08 16:56:08.741304 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.741275 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6czd\" (UniqueName: \"kubernetes.io/projected/d2bd1bab-bda5-45cf-8812-2a3e6b2026ba-kube-api-access-c6czd\") pod \"llmisvc-controller-manager-777c6bd7d5-q788k\" (UID: \"d2bd1bab-bda5-45cf-8812-2a3e6b2026ba\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q788k" Sep 08 16:56:08.911465 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.911367 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s8z86" Sep 08 16:56:08.921623 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:08.921600 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q788k" Sep 08 16:56:09.221402 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:09.221353 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-q788k"] Sep 08 16:56:09.226327 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:56:09.226302 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd2bd1bab_bda5_45cf_8812_2a3e6b2026ba.slice/crio-16ccb97a93178f1d4e1c55ef1d690209a17a1cc5bafbe0cd25e05c716b1b6632 WatchSource:0}: Error finding container 16ccb97a93178f1d4e1c55ef1d690209a17a1cc5bafbe0cd25e05c716b1b6632: Status 404 returned error can't find the container with id 16ccb97a93178f1d4e1c55ef1d690209a17a1cc5bafbe0cd25e05c716b1b6632 Sep 08 16:56:09.254797 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:56:09.254774 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517d507a_eadc_4cf7_9193_9b9bdfcefe66.slice/crio-37794ad1b8afd1cfc8dfc6d5bc6cc7d523f8f976aecd7a34b0088ec7a61ff2a9 WatchSource:0}: Error finding container 37794ad1b8afd1cfc8dfc6d5bc6cc7d523f8f976aecd7a34b0088ec7a61ff2a9: Status 404 returned error can't find the container with id 37794ad1b8afd1cfc8dfc6d5bc6cc7d523f8f976aecd7a34b0088ec7a61ff2a9 Sep 08 16:56:09.264256 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:09.264232 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-s8z86"] Sep 08 16:56:09.281086 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:09.281056 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q788k" event={"ID":"d2bd1bab-bda5-45cf-8812-2a3e6b2026ba","Type":"ContainerStarted","Data":"16ccb97a93178f1d4e1c55ef1d690209a17a1cc5bafbe0cd25e05c716b1b6632"} Sep 08 16:56:09.282259 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:09.282234 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s8z86" event={"ID":"517d507a-eadc-4cf7-9193-9b9bdfcefe66","Type":"ContainerStarted","Data":"37794ad1b8afd1cfc8dfc6d5bc6cc7d523f8f976aecd7a34b0088ec7a61ff2a9"} Sep 08 16:56:11.205093 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:11.205049 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:56:12.302591 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:12.302546 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s8z86" event={"ID":"517d507a-eadc-4cf7-9193-9b9bdfcefe66","Type":"ContainerStarted","Data":"e1778839852d58e34f978afa88edf5a3f259d869f27860bae069139c503b7721"} Sep 08 16:56:12.303060 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:12.302741 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s8z86" Sep 08 16:56:12.348813 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:12.348762 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s8z86" podStartSLOduration=2.058385842 podStartE2EDuration="4.348747722s" totalImagesPullingTime="2.29036188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:56:12.346653021 +0000 UTC m=+335.083484139" watchObservedRunningTime="2026-09-08 16:56:12.348747722 +0000 UTC m=+335.085578838" Sep 08 16:56:12.987599 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:12.987567 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j"] Sep 08 16:56:12.987809 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:12.987785 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" podUID="460d1a33-2dec-4313-9bb7-e5fe86e7effa" containerName="discovery" containerID="cri-o://10cacfd891971ea224a20f41c2a894744dfd46a26c5596be5e69b64e8bc68562" gracePeriod=30 Sep 08 16:56:13.219275 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.219250 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:56:13.261815 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.261787 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-csr-ca-configmap\") pod \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " Sep 08 16:56:13.261930 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.261840 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-kubeconfig\") pod \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " Sep 08 16:56:13.261930 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.261888 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-token\") pod \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " Sep 08 16:56:13.261930 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.261916 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7m9vn\" (UniqueName: \"kubernetes.io/projected/460d1a33-2dec-4313-9bb7-e5fe86e7effa-kube-api-access-7m9vn\") pod \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " Sep 08 16:56:13.262090 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.261954 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/460d1a33-2dec-4313-9bb7-e5fe86e7effa-cacerts\") pod \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " Sep 08 16:56:13.262090 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.262017 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/460d1a33-2dec-4313-9bb7-e5fe86e7effa-local-certs\") pod \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " Sep 08 16:56:13.262090 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.262066 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-csr-dns-cert\") pod \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\" (UID: \"460d1a33-2dec-4313-9bb7-e5fe86e7effa\") " Sep 08 16:56:13.262552 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.262230 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "460d1a33-2dec-4313-9bb7-e5fe86e7effa" (UID: "460d1a33-2dec-4313-9bb7-e5fe86e7effa"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:56:13.264322 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.264294 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/460d1a33-2dec-4313-9bb7-e5fe86e7effa-cacerts" (OuterVolumeSpecName: "cacerts") pod "460d1a33-2dec-4313-9bb7-e5fe86e7effa" (UID: "460d1a33-2dec-4313-9bb7-e5fe86e7effa"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:56:13.264439 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.264321 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "460d1a33-2dec-4313-9bb7-e5fe86e7effa" (UID: "460d1a33-2dec-4313-9bb7-e5fe86e7effa"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:56:13.264504 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.264468 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-token" (OuterVolumeSpecName: "istio-token") pod "460d1a33-2dec-4313-9bb7-e5fe86e7effa" (UID: "460d1a33-2dec-4313-9bb7-e5fe86e7effa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:56:13.264553 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.264521 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460d1a33-2dec-4313-9bb7-e5fe86e7effa-kube-api-access-7m9vn" (OuterVolumeSpecName: "kube-api-access-7m9vn") pod "460d1a33-2dec-4313-9bb7-e5fe86e7effa" (UID: "460d1a33-2dec-4313-9bb7-e5fe86e7effa"). InnerVolumeSpecName "kube-api-access-7m9vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:56:13.264553 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.264519 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "460d1a33-2dec-4313-9bb7-e5fe86e7effa" (UID: "460d1a33-2dec-4313-9bb7-e5fe86e7effa"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:56:13.264622 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.264606 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460d1a33-2dec-4313-9bb7-e5fe86e7effa-local-certs" (OuterVolumeSpecName: "local-certs") pod "460d1a33-2dec-4313-9bb7-e5fe86e7effa" (UID: "460d1a33-2dec-4313-9bb7-e5fe86e7effa"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:56:13.308492 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.308418 2565 generic.go:356] "Generic (PLEG): container finished" podID="460d1a33-2dec-4313-9bb7-e5fe86e7effa" containerID="10cacfd891971ea224a20f41c2a894744dfd46a26c5596be5e69b64e8bc68562" exitCode=0 Sep 08 16:56:13.308492 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.308482 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" Sep 08 16:56:13.308910 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.308525 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" event={"ID":"460d1a33-2dec-4313-9bb7-e5fe86e7effa","Type":"ContainerDied","Data":"10cacfd891971ea224a20f41c2a894744dfd46a26c5596be5e69b64e8bc68562"} Sep 08 16:56:13.308910 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.308576 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j" event={"ID":"460d1a33-2dec-4313-9bb7-e5fe86e7effa","Type":"ContainerDied","Data":"54d6409572f34edba8183e146b599e067874daa2f3e8a5b2ca1c3e64ef31c32b"} Sep 08 16:56:13.308910 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.308601 2565 scope.go:117] "RemoveContainer" containerID="10cacfd891971ea224a20f41c2a894744dfd46a26c5596be5e69b64e8bc68562" Sep 08 16:56:13.310545 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.310405 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q788k" event={"ID":"d2bd1bab-bda5-45cf-8812-2a3e6b2026ba","Type":"ContainerStarted","Data":"9b1560e6cec40323c6a2c7b9a595dd1fe21cc0d3256abed50f1c1f453d30ab5a"} Sep 08 16:56:13.310660 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.310638 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q788k" Sep 08 16:56:13.324726 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.324707 2565 scope.go:117] "RemoveContainer" containerID="10cacfd891971ea224a20f41c2a894744dfd46a26c5596be5e69b64e8bc68562" Sep 08 16:56:13.325041 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:56:13.325019 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10cacfd891971ea224a20f41c2a894744dfd46a26c5596be5e69b64e8bc68562\": container with ID starting with 10cacfd891971ea224a20f41c2a894744dfd46a26c5596be5e69b64e8bc68562 not found: ID does not exist" containerID="10cacfd891971ea224a20f41c2a894744dfd46a26c5596be5e69b64e8bc68562" Sep 08 16:56:13.325108 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.325047 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10cacfd891971ea224a20f41c2a894744dfd46a26c5596be5e69b64e8bc68562"} err="failed to get container status \"10cacfd891971ea224a20f41c2a894744dfd46a26c5596be5e69b64e8bc68562\": rpc error: code = NotFound desc = could not find container \"10cacfd891971ea224a20f41c2a894744dfd46a26c5596be5e69b64e8bc68562\": container with ID starting with 10cacfd891971ea224a20f41c2a894744dfd46a26c5596be5e69b64e8bc68562 not found: ID does not exist" Sep 08 16:56:13.362954 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.362927 2565 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/460d1a33-2dec-4313-9bb7-e5fe86e7effa-local-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:56:13.362954 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.362950 2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-csr-dns-cert\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:56:13.363112 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.363033 2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-csr-ca-configmap\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:56:13.363112 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.363045 2565 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-kubeconfig\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:56:13.363112 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.363059 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/460d1a33-2dec-4313-9bb7-e5fe86e7effa-istio-token\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:56:13.363112 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.363069 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m9vn\" (UniqueName: \"kubernetes.io/projected/460d1a33-2dec-4313-9bb7-e5fe86e7effa-kube-api-access-7m9vn\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:56:13.363112 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.363077 2565 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/460d1a33-2dec-4313-9bb7-e5fe86e7effa-cacerts\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:56:13.410329 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.410283 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q788k" podStartSLOduration=1.863351265 podStartE2EDuration="5.410271257s" totalImagesPullingTime="3.546919992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:56:13.409375112 +0000 UTC m=+336.146206229" watchObservedRunningTime="2026-09-08 16:56:13.410271257 +0000 UTC m=+336.147102372" Sep 08 16:56:13.460079 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.460054 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j"] Sep 08 16:56:13.485184 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.485157 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-c4z6j"] Sep 08 16:56:13.759639 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:13.759604 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="460d1a33-2dec-4313-9bb7-e5fe86e7effa" path="/var/lib/kubelet/pods/460d1a33-2dec-4313-9bb7-e5fe86e7effa/volumes" Sep 08 16:56:33.185623 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:33.185542 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:56:43.315250 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:43.315220 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s8z86" Sep 08 16:56:44.318811 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:44.318781 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q788k" Sep 08 16:56:56.545048 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:56.545009 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:56:56.710593 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:56:56.710553 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:03.270944 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.270908 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5"] Sep 08 16:57:03.271312 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.271216 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="460d1a33-2dec-4313-9bb7-e5fe86e7effa" containerName="discovery" Sep 08 16:57:03.271312 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.271227 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="460d1a33-2dec-4313-9bb7-e5fe86e7effa" containerName="discovery" Sep 08 16:57:03.271312 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.271304 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="460d1a33-2dec-4313-9bb7-e5fe86e7effa" containerName="discovery" Sep 08 16:57:03.274308 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.274291 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5" Sep 08 16:57:03.277600 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.277579 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 16:57:03.277712 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.277616 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-v8k2r\"" Sep 08 16:57:03.278455 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.278437 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 16:57:03.294349 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.294326 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5"] Sep 08 16:57:03.360202 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.360178 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t"] Sep 08 16:57:03.363573 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.363557 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t" Sep 08 16:57:03.375389 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.375368 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t"] Sep 08 16:57:03.438945 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.438921 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/668b2d21-1dca-47b8-aab4-b1d9dbb3d62f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t\" (UID: \"668b2d21-1dca-47b8-aab4-b1d9dbb3d62f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t" Sep 08 16:57:03.439044 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.438952 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/668b2d21-1dca-47b8-aab4-b1d9dbb3d62f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t\" (UID: \"668b2d21-1dca-47b8-aab4-b1d9dbb3d62f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t" Sep 08 16:57:03.439044 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.438974 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14098b05-4970-40a0-89ca-35090c44322f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5\" (UID: \"14098b05-4970-40a0-89ca-35090c44322f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5" Sep 08 16:57:03.439115 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.439092 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzwzt\" (UniqueName: \"kubernetes.io/projected/14098b05-4970-40a0-89ca-35090c44322f-kube-api-access-wzwzt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5\" (UID: \"14098b05-4970-40a0-89ca-35090c44322f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5" Sep 08 16:57:03.439150 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.439129 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5rm8\" (UniqueName: \"kubernetes.io/projected/668b2d21-1dca-47b8-aab4-b1d9dbb3d62f-kube-api-access-j5rm8\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t\" (UID: \"668b2d21-1dca-47b8-aab4-b1d9dbb3d62f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t" Sep 08 16:57:03.439183 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.439152 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14098b05-4970-40a0-89ca-35090c44322f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5\" (UID: \"14098b05-4970-40a0-89ca-35090c44322f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5" Sep 08 16:57:03.456001 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.455976 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4"] Sep 08 16:57:03.459204 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.459186 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4" Sep 08 16:57:03.472924 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.472899 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4"] Sep 08 16:57:03.536925 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.536861 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9"] Sep 08 16:57:03.540105 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.540087 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9" Sep 08 16:57:03.540275 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.540253 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2396a593-673d-4b29-9317-a42cb4ab28c5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4\" (UID: \"2396a593-673d-4b29-9317-a42cb4ab28c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4" Sep 08 16:57:03.540358 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.540284 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2396a593-673d-4b29-9317-a42cb4ab28c5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4\" (UID: \"2396a593-673d-4b29-9317-a42cb4ab28c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4" Sep 08 16:57:03.540417 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.540376 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt8sr\" (UniqueName: \"kubernetes.io/projected/2396a593-673d-4b29-9317-a42cb4ab28c5-kube-api-access-lt8sr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4\" (UID: \"2396a593-673d-4b29-9317-a42cb4ab28c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4" Sep 08 16:57:03.540485 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.540450 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzwzt\" (UniqueName: \"kubernetes.io/projected/14098b05-4970-40a0-89ca-35090c44322f-kube-api-access-wzwzt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5\" (UID: \"14098b05-4970-40a0-89ca-35090c44322f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5" Sep 08 16:57:03.540537 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.540493 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5rm8\" (UniqueName: \"kubernetes.io/projected/668b2d21-1dca-47b8-aab4-b1d9dbb3d62f-kube-api-access-j5rm8\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t\" (UID: \"668b2d21-1dca-47b8-aab4-b1d9dbb3d62f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t" Sep 08 16:57:03.540537 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.540516 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14098b05-4970-40a0-89ca-35090c44322f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5\" (UID: \"14098b05-4970-40a0-89ca-35090c44322f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5" Sep 08 16:57:03.540636 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.540567 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/668b2d21-1dca-47b8-aab4-b1d9dbb3d62f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t\" (UID: \"668b2d21-1dca-47b8-aab4-b1d9dbb3d62f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t" Sep 08 16:57:03.540636 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.540591 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/668b2d21-1dca-47b8-aab4-b1d9dbb3d62f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t\" (UID: \"668b2d21-1dca-47b8-aab4-b1d9dbb3d62f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t" Sep 08 16:57:03.540636 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.540618 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14098b05-4970-40a0-89ca-35090c44322f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5\" (UID: \"14098b05-4970-40a0-89ca-35090c44322f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5" Sep 08 16:57:03.540941 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.540920 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14098b05-4970-40a0-89ca-35090c44322f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5\" (UID: \"14098b05-4970-40a0-89ca-35090c44322f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5" Sep 08 16:57:03.541048 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.541027 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/668b2d21-1dca-47b8-aab4-b1d9dbb3d62f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t\" (UID: \"668b2d21-1dca-47b8-aab4-b1d9dbb3d62f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t" Sep 08 16:57:03.541114 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.541100 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/668b2d21-1dca-47b8-aab4-b1d9dbb3d62f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t\" (UID: \"668b2d21-1dca-47b8-aab4-b1d9dbb3d62f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t" Sep 08 16:57:03.541151 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.541107 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14098b05-4970-40a0-89ca-35090c44322f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5\" (UID: \"14098b05-4970-40a0-89ca-35090c44322f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5" Sep 08 16:57:03.552603 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.552565 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5rm8\" (UniqueName: \"kubernetes.io/projected/668b2d21-1dca-47b8-aab4-b1d9dbb3d62f-kube-api-access-j5rm8\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t\" (UID: \"668b2d21-1dca-47b8-aab4-b1d9dbb3d62f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t" Sep 08 16:57:03.553094 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.553075 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9"] Sep 08 16:57:03.553545 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.553524 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzwzt\" (UniqueName: \"kubernetes.io/projected/14098b05-4970-40a0-89ca-35090c44322f-kube-api-access-wzwzt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5\" (UID: \"14098b05-4970-40a0-89ca-35090c44322f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5" Sep 08 16:57:03.583207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.583185 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5" Sep 08 16:57:03.642071 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.642036 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9\" (UID: \"ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9" Sep 08 16:57:03.642182 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.642122 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9\" (UID: \"ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9" Sep 08 16:57:03.642182 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.642163 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrwp2\" (UniqueName: \"kubernetes.io/projected/ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5-kube-api-access-wrwp2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9\" (UID: \"ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9" Sep 08 16:57:03.642311 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.642196 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2396a593-673d-4b29-9317-a42cb4ab28c5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4\" (UID: \"2396a593-673d-4b29-9317-a42cb4ab28c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4" Sep 08 16:57:03.642311 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.642227 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2396a593-673d-4b29-9317-a42cb4ab28c5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4\" (UID: \"2396a593-673d-4b29-9317-a42cb4ab28c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4" Sep 08 16:57:03.642311 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.642291 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt8sr\" (UniqueName: \"kubernetes.io/projected/2396a593-673d-4b29-9317-a42cb4ab28c5-kube-api-access-lt8sr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4\" (UID: \"2396a593-673d-4b29-9317-a42cb4ab28c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4" Sep 08 16:57:03.642886 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.642859 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2396a593-673d-4b29-9317-a42cb4ab28c5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4\" (UID: \"2396a593-673d-4b29-9317-a42cb4ab28c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4" Sep 08 16:57:03.643514 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.643488 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2396a593-673d-4b29-9317-a42cb4ab28c5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4\" (UID: \"2396a593-673d-4b29-9317-a42cb4ab28c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4" Sep 08 16:57:03.652332 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.652294 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt8sr\" (UniqueName: \"kubernetes.io/projected/2396a593-673d-4b29-9317-a42cb4ab28c5-kube-api-access-lt8sr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4\" (UID: \"2396a593-673d-4b29-9317-a42cb4ab28c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4" Sep 08 16:57:03.671907 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.671889 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t" Sep 08 16:57:03.742853 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.742827 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9\" (UID: \"ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9" Sep 08 16:57:03.742954 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.742875 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9\" (UID: \"ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9" Sep 08 16:57:03.742954 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.742902 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrwp2\" (UniqueName: \"kubernetes.io/projected/ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5-kube-api-access-wrwp2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9\" (UID: \"ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9" Sep 08 16:57:03.743287 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.743261 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9\" (UID: \"ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9" Sep 08 16:57:03.743385 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.743268 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9\" (UID: \"ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9" Sep 08 16:57:03.757924 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.757894 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrwp2\" (UniqueName: \"kubernetes.io/projected/ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5-kube-api-access-wrwp2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9\" (UID: \"ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9" Sep 08 16:57:03.767433 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.767391 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4" Sep 08 16:57:03.849973 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.849942 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9" Sep 08 16:57:03.864088 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.864060 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5"] Sep 08 16:57:03.866571 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:57:03.866541 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14098b05_4970_40a0_89ca_35090c44322f.slice/crio-a6fce9a51efe07469034bdf3e382073b9ff59210779111087f43f4abea3b8b33 WatchSource:0}: Error finding container a6fce9a51efe07469034bdf3e382073b9ff59210779111087f43f4abea3b8b33: Status 404 returned error can't find the container with id a6fce9a51efe07469034bdf3e382073b9ff59210779111087f43f4abea3b8b33 Sep 08 16:57:03.963935 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:03.963902 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t"] Sep 08 16:57:03.967674 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:57:03.967643 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668b2d21_1dca_47b8_aab4_b1d9dbb3d62f.slice/crio-d626366d9af6f3cdb13e369a8a5c14e3a43b5ccd8ecc0f3489a71491656efddf WatchSource:0}: Error finding container d626366d9af6f3cdb13e369a8a5c14e3a43b5ccd8ecc0f3489a71491656efddf: Status 404 returned error can't find the container with id d626366d9af6f3cdb13e369a8a5c14e3a43b5ccd8ecc0f3489a71491656efddf Sep 08 16:57:04.077670 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:04.077639 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4"] Sep 08 16:57:04.077789 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:57:04.077666 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2396a593_673d_4b29_9317_a42cb4ab28c5.slice/crio-9ceec90044f14be3854afbc345bdc1bac435a666696f556507783aa4fe8efdea WatchSource:0}: Error finding container 9ceec90044f14be3854afbc345bdc1bac435a666696f556507783aa4fe8efdea: Status 404 returned error can't find the container with id 9ceec90044f14be3854afbc345bdc1bac435a666696f556507783aa4fe8efdea Sep 08 16:57:04.134882 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:04.134860 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9"] Sep 08 16:57:04.152282 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:57:04.152259 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5bf2d6_fcc5_4581_8be1_7a3b23ffc4f5.slice/crio-ede7ddb72a5f6e38143aaef990936c09d032466838b360e4a753e9f81d37c911 WatchSource:0}: Error finding container ede7ddb72a5f6e38143aaef990936c09d032466838b360e4a753e9f81d37c911: Status 404 returned error can't find the container with id ede7ddb72a5f6e38143aaef990936c09d032466838b360e4a753e9f81d37c911 Sep 08 16:57:04.551859 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:04.551826 2565 generic.go:356] "Generic (PLEG): container finished" podID="14098b05-4970-40a0-89ca-35090c44322f" containerID="acf3c573e0d14f1f74a3ea5bd03e317391959d0eb4cd357d32f3c7ac5663182c" exitCode=0 Sep 08 16:57:04.552267 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:04.551917 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5" event={"ID":"14098b05-4970-40a0-89ca-35090c44322f","Type":"ContainerDied","Data":"acf3c573e0d14f1f74a3ea5bd03e317391959d0eb4cd357d32f3c7ac5663182c"} Sep 08 16:57:04.552267 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:04.551985 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5" event={"ID":"14098b05-4970-40a0-89ca-35090c44322f","Type":"ContainerStarted","Data":"a6fce9a51efe07469034bdf3e382073b9ff59210779111087f43f4abea3b8b33"} Sep 08 16:57:04.553645 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:04.553616 2565 generic.go:356] "Generic (PLEG): container finished" podID="2396a593-673d-4b29-9317-a42cb4ab28c5" containerID="07ef914ab43b29f422166b2267e7094e5eaa7b60ccb7a5ff75b9c93bc37f397e" exitCode=0 Sep 08 16:57:04.553956 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:04.553650 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4" event={"ID":"2396a593-673d-4b29-9317-a42cb4ab28c5","Type":"ContainerDied","Data":"07ef914ab43b29f422166b2267e7094e5eaa7b60ccb7a5ff75b9c93bc37f397e"} Sep 08 16:57:04.553956 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:04.553700 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4" event={"ID":"2396a593-673d-4b29-9317-a42cb4ab28c5","Type":"ContainerStarted","Data":"9ceec90044f14be3854afbc345bdc1bac435a666696f556507783aa4fe8efdea"} Sep 08 16:57:04.555365 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:04.555344 2565 generic.go:356] "Generic (PLEG): container finished" podID="ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5" containerID="15df67bbf07bf20e7bd9479d5b9a3c095de0636712368815b983e787f4138517" exitCode=0 Sep 08 16:57:04.555469 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:04.555450 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9" event={"ID":"ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5","Type":"ContainerDied","Data":"15df67bbf07bf20e7bd9479d5b9a3c095de0636712368815b983e787f4138517"} Sep 08 16:57:04.555535 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:04.555496 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9" event={"ID":"ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5","Type":"ContainerStarted","Data":"ede7ddb72a5f6e38143aaef990936c09d032466838b360e4a753e9f81d37c911"} Sep 08 16:57:04.557154 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:04.557133 2565 generic.go:356] "Generic (PLEG): container finished" podID="668b2d21-1dca-47b8-aab4-b1d9dbb3d62f" containerID="8e1fa11d45a5421e75bb5b8b6af6bae3f2e1468d859afa4504470e56fe9adc9e" exitCode=0 Sep 08 16:57:04.557242 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:04.557201 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t" event={"ID":"668b2d21-1dca-47b8-aab4-b1d9dbb3d62f","Type":"ContainerDied","Data":"8e1fa11d45a5421e75bb5b8b6af6bae3f2e1468d859afa4504470e56fe9adc9e"} Sep 08 16:57:04.557312 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:04.557241 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t" event={"ID":"668b2d21-1dca-47b8-aab4-b1d9dbb3d62f","Type":"ContainerStarted","Data":"d626366d9af6f3cdb13e369a8a5c14e3a43b5ccd8ecc0f3489a71491656efddf"} Sep 08 16:57:05.564540 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:05.564506 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4" event={"ID":"2396a593-673d-4b29-9317-a42cb4ab28c5","Type":"ContainerStarted","Data":"89b04ea11a2f005d17e06bb901058fe525f2b7964f6d4145f1a4fcd02ee0ecfa"} Sep 08 16:57:06.571216 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:06.571188 2565 generic.go:356] "Generic (PLEG): container finished" podID="2396a593-673d-4b29-9317-a42cb4ab28c5" containerID="89b04ea11a2f005d17e06bb901058fe525f2b7964f6d4145f1a4fcd02ee0ecfa" exitCode=0 Sep 08 16:57:06.571583 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:06.571268 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4" event={"ID":"2396a593-673d-4b29-9317-a42cb4ab28c5","Type":"ContainerDied","Data":"89b04ea11a2f005d17e06bb901058fe525f2b7964f6d4145f1a4fcd02ee0ecfa"} Sep 08 16:57:06.573530 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:06.573498 2565 generic.go:356] "Generic (PLEG): container finished" podID="ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5" containerID="ff54a06d1a8efccd105acf06224480090f839333fa599b74acdd1d1520f223e4" exitCode=0 Sep 08 16:57:06.573628 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:06.573532 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9" event={"ID":"ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5","Type":"ContainerDied","Data":"ff54a06d1a8efccd105acf06224480090f839333fa599b74acdd1d1520f223e4"} Sep 08 16:57:06.575747 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:06.575729 2565 generic.go:356] "Generic (PLEG): container finished" podID="14098b05-4970-40a0-89ca-35090c44322f" containerID="1d9dc8d1195e24d7a94c6f309f4db5f06a90f90181cb0fb7816c30b854599b54" exitCode=0 Sep 08 16:57:06.575835 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:06.575755 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5" event={"ID":"14098b05-4970-40a0-89ca-35090c44322f","Type":"ContainerDied","Data":"1d9dc8d1195e24d7a94c6f309f4db5f06a90f90181cb0fb7816c30b854599b54"} Sep 08 16:57:07.582437 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:07.582395 2565 generic.go:356] "Generic (PLEG): container finished" podID="14098b05-4970-40a0-89ca-35090c44322f" containerID="7d3c1d477624db90eedea00c636eccbe78000a93d91ed8563c546985b16455e4" exitCode=0 Sep 08 16:57:07.582834 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:07.582487 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5" event={"ID":"14098b05-4970-40a0-89ca-35090c44322f","Type":"ContainerDied","Data":"7d3c1d477624db90eedea00c636eccbe78000a93d91ed8563c546985b16455e4"} Sep 08 16:57:07.584543 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:07.584515 2565 generic.go:356] "Generic (PLEG): container finished" podID="2396a593-673d-4b29-9317-a42cb4ab28c5" containerID="f8c49a1ec752994a40131c4d7c7613416d3895f4731b714c49e4a54db699d158" exitCode=0 Sep 08 16:57:07.584658 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:07.584604 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4" event={"ID":"2396a593-673d-4b29-9317-a42cb4ab28c5","Type":"ContainerDied","Data":"f8c49a1ec752994a40131c4d7c7613416d3895f4731b714c49e4a54db699d158"} Sep 08 16:57:07.586688 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:07.586665 2565 generic.go:356] "Generic (PLEG): container finished" podID="ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5" containerID="e20f7cca5a55b7adc1428217df8b88b9109e6985482e191b8c7e2e3700065a2f" exitCode=0 Sep 08 16:57:07.586789 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:07.586736 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9" event={"ID":"ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5","Type":"ContainerDied","Data":"e20f7cca5a55b7adc1428217df8b88b9109e6985482e191b8c7e2e3700065a2f"} Sep 08 16:57:07.588311 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:07.588289 2565 generic.go:356] "Generic (PLEG): container finished" podID="668b2d21-1dca-47b8-aab4-b1d9dbb3d62f" containerID="8606f6794f67f810df0b34c4791d12b3d29e906b8cf21b65dccac4482ca31efa" exitCode=0 Sep 08 16:57:07.588402 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:07.588333 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t" event={"ID":"668b2d21-1dca-47b8-aab4-b1d9dbb3d62f","Type":"ContainerDied","Data":"8606f6794f67f810df0b34c4791d12b3d29e906b8cf21b65dccac4482ca31efa"} Sep 08 16:57:08.595400 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.595360 2565 generic.go:356] "Generic (PLEG): container finished" podID="668b2d21-1dca-47b8-aab4-b1d9dbb3d62f" containerID="99406f1c9ae6d3008f29e00afd098db9e073b5b66e90c751f61a5c001debdb28" exitCode=0 Sep 08 16:57:08.595759 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.595460 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t" event={"ID":"668b2d21-1dca-47b8-aab4-b1d9dbb3d62f","Type":"ContainerDied","Data":"99406f1c9ae6d3008f29e00afd098db9e073b5b66e90c751f61a5c001debdb28"} Sep 08 16:57:08.745968 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.745943 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5" Sep 08 16:57:08.784604 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.784507 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9" Sep 08 16:57:08.788461 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.788441 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4" Sep 08 16:57:08.886843 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.886777 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14098b05-4970-40a0-89ca-35090c44322f-bundle\") pod \"14098b05-4970-40a0-89ca-35090c44322f\" (UID: \"14098b05-4970-40a0-89ca-35090c44322f\") " Sep 08 16:57:08.886843 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.886827 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrwp2\" (UniqueName: \"kubernetes.io/projected/ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5-kube-api-access-wrwp2\") pod \"ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5\" (UID: \"ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5\") " Sep 08 16:57:08.887005 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.886853 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5-bundle\") pod \"ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5\" (UID: \"ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5\") " Sep 08 16:57:08.887005 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.886887 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzwzt\" (UniqueName: \"kubernetes.io/projected/14098b05-4970-40a0-89ca-35090c44322f-kube-api-access-wzwzt\") pod \"14098b05-4970-40a0-89ca-35090c44322f\" (UID: \"14098b05-4970-40a0-89ca-35090c44322f\") " Sep 08 16:57:08.887112 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.887049 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14098b05-4970-40a0-89ca-35090c44322f-util\") pod \"14098b05-4970-40a0-89ca-35090c44322f\" (UID: \"14098b05-4970-40a0-89ca-35090c44322f\") " Sep 08 16:57:08.887112 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.887090 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2396a593-673d-4b29-9317-a42cb4ab28c5-bundle\") pod \"2396a593-673d-4b29-9317-a42cb4ab28c5\" (UID: \"2396a593-673d-4b29-9317-a42cb4ab28c5\") " Sep 08 16:57:08.887213 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.887129 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt8sr\" (UniqueName: \"kubernetes.io/projected/2396a593-673d-4b29-9317-a42cb4ab28c5-kube-api-access-lt8sr\") pod \"2396a593-673d-4b29-9317-a42cb4ab28c5\" (UID: \"2396a593-673d-4b29-9317-a42cb4ab28c5\") " Sep 08 16:57:08.887213 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.887161 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2396a593-673d-4b29-9317-a42cb4ab28c5-util\") pod \"2396a593-673d-4b29-9317-a42cb4ab28c5\" (UID: \"2396a593-673d-4b29-9317-a42cb4ab28c5\") " Sep 08 16:57:08.887213 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.887185 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5-util\") pod \"ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5\" (UID: \"ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5\") " Sep 08 16:57:08.887686 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.887661 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14098b05-4970-40a0-89ca-35090c44322f-bundle" (OuterVolumeSpecName: "bundle") pod "14098b05-4970-40a0-89ca-35090c44322f" (UID: "14098b05-4970-40a0-89ca-35090c44322f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:57:08.888166 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.888124 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5-bundle" (OuterVolumeSpecName: "bundle") pod "ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5" (UID: "ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:57:08.888259 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.888205 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2396a593-673d-4b29-9317-a42cb4ab28c5-bundle" (OuterVolumeSpecName: "bundle") pod "2396a593-673d-4b29-9317-a42cb4ab28c5" (UID: "2396a593-673d-4b29-9317-a42cb4ab28c5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:57:08.889584 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.889551 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5-kube-api-access-wrwp2" (OuterVolumeSpecName: "kube-api-access-wrwp2") pod "ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5" (UID: "ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5"). InnerVolumeSpecName "kube-api-access-wrwp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:57:08.889686 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.889649 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14098b05-4970-40a0-89ca-35090c44322f-kube-api-access-wzwzt" (OuterVolumeSpecName: "kube-api-access-wzwzt") pod "14098b05-4970-40a0-89ca-35090c44322f" (UID: "14098b05-4970-40a0-89ca-35090c44322f"). InnerVolumeSpecName "kube-api-access-wzwzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:57:08.890140 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.890113 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2396a593-673d-4b29-9317-a42cb4ab28c5-kube-api-access-lt8sr" (OuterVolumeSpecName: "kube-api-access-lt8sr") pod "2396a593-673d-4b29-9317-a42cb4ab28c5" (UID: "2396a593-673d-4b29-9317-a42cb4ab28c5"). InnerVolumeSpecName "kube-api-access-lt8sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:57:08.894248 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.894147 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14098b05-4970-40a0-89ca-35090c44322f-util" (OuterVolumeSpecName: "util") pod "14098b05-4970-40a0-89ca-35090c44322f" (UID: "14098b05-4970-40a0-89ca-35090c44322f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:57:08.896925 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.896887 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2396a593-673d-4b29-9317-a42cb4ab28c5-util" (OuterVolumeSpecName: "util") pod "2396a593-673d-4b29-9317-a42cb4ab28c5" (UID: "2396a593-673d-4b29-9317-a42cb4ab28c5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:57:08.897398 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.897381 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5-util" (OuterVolumeSpecName: "util") pod "ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5" (UID: "ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:57:08.988742 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.988720 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14098b05-4970-40a0-89ca-35090c44322f-bundle\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:57:08.988742 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.988742 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrwp2\" (UniqueName: \"kubernetes.io/projected/ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5-kube-api-access-wrwp2\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:57:08.988855 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.988752 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5-bundle\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:57:08.988855 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.988761 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzwzt\" (UniqueName: \"kubernetes.io/projected/14098b05-4970-40a0-89ca-35090c44322f-kube-api-access-wzwzt\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:57:08.988855 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.988769 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14098b05-4970-40a0-89ca-35090c44322f-util\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:57:08.988855 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.988777 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2396a593-673d-4b29-9317-a42cb4ab28c5-bundle\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:57:08.988855 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.988785 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt8sr\" (UniqueName: \"kubernetes.io/projected/2396a593-673d-4b29-9317-a42cb4ab28c5-kube-api-access-lt8sr\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:57:08.988855 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.988793 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2396a593-673d-4b29-9317-a42cb4ab28c5-util\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:57:08.988855 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:08.988801 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5-util\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:57:09.603466 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:09.603417 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4" Sep 08 16:57:09.603466 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:09.603443 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jl7d4" event={"ID":"2396a593-673d-4b29-9317-a42cb4ab28c5","Type":"ContainerDied","Data":"9ceec90044f14be3854afbc345bdc1bac435a666696f556507783aa4fe8efdea"} Sep 08 16:57:09.603932 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:09.603492 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ceec90044f14be3854afbc345bdc1bac435a666696f556507783aa4fe8efdea" Sep 08 16:57:09.605372 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:09.605353 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9" Sep 08 16:57:09.605512 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:09.605391 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akkmm9" event={"ID":"ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5","Type":"ContainerDied","Data":"ede7ddb72a5f6e38143aaef990936c09d032466838b360e4a753e9f81d37c911"} Sep 08 16:57:09.605512 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:09.605464 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ede7ddb72a5f6e38143aaef990936c09d032466838b360e4a753e9f81d37c911" Sep 08 16:57:09.607557 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:09.607539 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5" Sep 08 16:57:09.607666 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:09.607571 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwfhx5" event={"ID":"14098b05-4970-40a0-89ca-35090c44322f","Type":"ContainerDied","Data":"a6fce9a51efe07469034bdf3e382073b9ff59210779111087f43f4abea3b8b33"} Sep 08 16:57:09.607666 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:09.607609 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6fce9a51efe07469034bdf3e382073b9ff59210779111087f43f4abea3b8b33" Sep 08 16:57:09.741029 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:09.741006 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t" Sep 08 16:57:09.895015 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:09.894933 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/668b2d21-1dca-47b8-aab4-b1d9dbb3d62f-util\") pod \"668b2d21-1dca-47b8-aab4-b1d9dbb3d62f\" (UID: \"668b2d21-1dca-47b8-aab4-b1d9dbb3d62f\") " Sep 08 16:57:09.895015 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:09.894985 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5rm8\" (UniqueName: \"kubernetes.io/projected/668b2d21-1dca-47b8-aab4-b1d9dbb3d62f-kube-api-access-j5rm8\") pod \"668b2d21-1dca-47b8-aab4-b1d9dbb3d62f\" (UID: \"668b2d21-1dca-47b8-aab4-b1d9dbb3d62f\") " Sep 08 16:57:09.895221 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:09.895062 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/668b2d21-1dca-47b8-aab4-b1d9dbb3d62f-bundle\") pod \"668b2d21-1dca-47b8-aab4-b1d9dbb3d62f\" (UID: \"668b2d21-1dca-47b8-aab4-b1d9dbb3d62f\") " Sep 08 16:57:09.895581 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:09.895549 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668b2d21-1dca-47b8-aab4-b1d9dbb3d62f-bundle" (OuterVolumeSpecName: "bundle") pod "668b2d21-1dca-47b8-aab4-b1d9dbb3d62f" (UID: "668b2d21-1dca-47b8-aab4-b1d9dbb3d62f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:57:09.897211 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:09.897187 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668b2d21-1dca-47b8-aab4-b1d9dbb3d62f-kube-api-access-j5rm8" (OuterVolumeSpecName: "kube-api-access-j5rm8") pod "668b2d21-1dca-47b8-aab4-b1d9dbb3d62f" (UID: "668b2d21-1dca-47b8-aab4-b1d9dbb3d62f"). InnerVolumeSpecName "kube-api-access-j5rm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:57:09.901142 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:09.901101 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668b2d21-1dca-47b8-aab4-b1d9dbb3d62f-util" (OuterVolumeSpecName: "util") pod "668b2d21-1dca-47b8-aab4-b1d9dbb3d62f" (UID: "668b2d21-1dca-47b8-aab4-b1d9dbb3d62f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:57:09.996089 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:09.996062 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/668b2d21-1dca-47b8-aab4-b1d9dbb3d62f-bundle\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:57:09.996089 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:09.996084 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/668b2d21-1dca-47b8-aab4-b1d9dbb3d62f-util\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:57:09.996089 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:09.996093 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5rm8\" (UniqueName: \"kubernetes.io/projected/668b2d21-1dca-47b8-aab4-b1d9dbb3d62f-kube-api-access-j5rm8\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:57:10.622404 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:10.622366 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t" Sep 08 16:57:10.622828 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:10.622366 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pwh8t" event={"ID":"668b2d21-1dca-47b8-aab4-b1d9dbb3d62f","Type":"ContainerDied","Data":"d626366d9af6f3cdb13e369a8a5c14e3a43b5ccd8ecc0f3489a71491656efddf"} Sep 08 16:57:10.622828 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:10.622495 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d626366d9af6f3cdb13e369a8a5c14e3a43b5ccd8ecc0f3489a71491656efddf" Sep 08 16:57:16.106927 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:16.106840 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:18.329171 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329135 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l"] Sep 08 16:57:18.329556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329449 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14098b05-4970-40a0-89ca-35090c44322f" containerName="pull" Sep 08 16:57:18.329556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329461 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="14098b05-4970-40a0-89ca-35090c44322f" containerName="pull" Sep 08 16:57:18.329556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329469 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="668b2d21-1dca-47b8-aab4-b1d9dbb3d62f" containerName="pull" Sep 08 16:57:18.329556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329475 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="668b2d21-1dca-47b8-aab4-b1d9dbb3d62f" containerName="pull" Sep 08 16:57:18.329556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329483 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14098b05-4970-40a0-89ca-35090c44322f" containerName="util" Sep 08 16:57:18.329556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329491 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="14098b05-4970-40a0-89ca-35090c44322f" containerName="util" Sep 08 16:57:18.329556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329501 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14098b05-4970-40a0-89ca-35090c44322f" containerName="extract" Sep 08 16:57:18.329556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329506 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="14098b05-4970-40a0-89ca-35090c44322f" containerName="extract" Sep 08 16:57:18.329556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329515 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5" containerName="util" Sep 08 16:57:18.329556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329520 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5" containerName="util" Sep 08 16:57:18.329556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329525 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2396a593-673d-4b29-9317-a42cb4ab28c5" containerName="extract" Sep 08 16:57:18.329556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329530 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2396a593-673d-4b29-9317-a42cb4ab28c5" containerName="extract" Sep 08 16:57:18.329556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329536 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5" containerName="extract" Sep 08 16:57:18.329556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329540 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5" containerName="extract" Sep 08 16:57:18.329556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329547 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="668b2d21-1dca-47b8-aab4-b1d9dbb3d62f" containerName="extract" Sep 08 16:57:18.329556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329554 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="668b2d21-1dca-47b8-aab4-b1d9dbb3d62f" containerName="extract" Sep 08 16:57:18.329556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329560 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2396a593-673d-4b29-9317-a42cb4ab28c5" containerName="util" Sep 08 16:57:18.329556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329565 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2396a593-673d-4b29-9317-a42cb4ab28c5" containerName="util" Sep 08 16:57:18.330136 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329571 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2396a593-673d-4b29-9317-a42cb4ab28c5" containerName="pull" Sep 08 16:57:18.330136 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329577 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2396a593-673d-4b29-9317-a42cb4ab28c5" containerName="pull" Sep 08 16:57:18.330136 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329584 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="668b2d21-1dca-47b8-aab4-b1d9dbb3d62f" containerName="util" Sep 08 16:57:18.330136 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329589 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="668b2d21-1dca-47b8-aab4-b1d9dbb3d62f" containerName="util" Sep 08 16:57:18.330136 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329594 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5" containerName="pull" Sep 08 16:57:18.330136 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329599 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5" containerName="pull" Sep 08 16:57:18.330136 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329647 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed5bf2d6-fcc5-4581-8be1-7a3b23ffc4f5" containerName="extract" Sep 08 16:57:18.330136 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329656 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="14098b05-4970-40a0-89ca-35090c44322f" containerName="extract" Sep 08 16:57:18.330136 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329663 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2396a593-673d-4b29-9317-a42cb4ab28c5" containerName="extract" Sep 08 16:57:18.330136 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.329667 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="668b2d21-1dca-47b8-aab4-b1d9dbb3d62f" containerName="extract" Sep 08 16:57:18.332780 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.332766 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l" Sep 08 16:57:18.335833 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.335810 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 08 16:57:18.335950 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.335870 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-whjv6\"" Sep 08 16:57:18.336490 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.336473 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 08 16:57:18.354217 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.354196 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l"] Sep 08 16:57:18.357447 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.357411 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pblb4\" (UniqueName: \"kubernetes.io/projected/c15d6121-3965-4732-9abc-3b1e2ca9b3c8-kube-api-access-pblb4\") pod \"limitador-operator-controller-manager-74495c69f8-s242l\" (UID: \"c15d6121-3965-4732-9abc-3b1e2ca9b3c8\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l" Sep 08 16:57:18.457844 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.457812 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pblb4\" (UniqueName: \"kubernetes.io/projected/c15d6121-3965-4732-9abc-3b1e2ca9b3c8-kube-api-access-pblb4\") pod \"limitador-operator-controller-manager-74495c69f8-s242l\" (UID: \"c15d6121-3965-4732-9abc-3b1e2ca9b3c8\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l" Sep 08 16:57:18.474620 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.474598 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pblb4\" (UniqueName: \"kubernetes.io/projected/c15d6121-3965-4732-9abc-3b1e2ca9b3c8-kube-api-access-pblb4\") pod \"limitador-operator-controller-manager-74495c69f8-s242l\" (UID: \"c15d6121-3965-4732-9abc-3b1e2ca9b3c8\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l" Sep 08 16:57:18.642858 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.642790 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l" Sep 08 16:57:18.667911 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.667882 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:18.950807 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:57:18.950779 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15d6121_3965_4732_9abc_3b1e2ca9b3c8.slice/crio-31cbbf2dc0a1d25bbefcaf1c02c82e0af46a36535ee1f4c06e9c8ac07ca5a018 WatchSource:0}: Error finding container 31cbbf2dc0a1d25bbefcaf1c02c82e0af46a36535ee1f4c06e9c8ac07ca5a018: Status 404 returned error can't find the container with id 31cbbf2dc0a1d25bbefcaf1c02c82e0af46a36535ee1f4c06e9c8ac07ca5a018 Sep 08 16:57:18.951241 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:18.951216 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l"] Sep 08 16:57:19.091858 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:19.091821 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:19.669990 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:19.669953 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l" event={"ID":"c15d6121-3965-4732-9abc-3b1e2ca9b3c8","Type":"ContainerStarted","Data":"31cbbf2dc0a1d25bbefcaf1c02c82e0af46a36535ee1f4c06e9c8ac07ca5a018"} Sep 08 16:57:20.454727 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:20.454678 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:20.697210 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:20.697099 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:21.682706 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:21.682669 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l" event={"ID":"c15d6121-3965-4732-9abc-3b1e2ca9b3c8","Type":"ContainerStarted","Data":"87ee02417f4f290254debb9c789def78497d2a90337eebd39891b2920da6a63e"} Sep 08 16:57:21.682853 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:21.682815 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l" Sep 08 16:57:21.718040 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:21.717993 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l" podStartSLOduration=1.268317526 podStartE2EDuration="3.717980407s" totalImagesPullingTime="2.449662881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:57:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:57:21.717513221 +0000 UTC m=+404.454344337" watchObservedRunningTime="2026-09-08 16:57:21.717980407 +0000 UTC m=+404.454811523" Sep 08 16:57:21.864486 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:21.864447 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:22.018270 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:22.018188 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:22.145416 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:22.145385 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-x9b85"] Sep 08 16:57:22.148797 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:22.148779 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-x9b85" Sep 08 16:57:22.155089 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:22.155060 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-g4w4s\"" Sep 08 16:57:22.168705 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:22.168682 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-x9b85"] Sep 08 16:57:22.193149 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:22.193123 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njx27\" (UniqueName: \"kubernetes.io/projected/b3c4cd20-2303-48e9-9c7b-dd655487489e-kube-api-access-njx27\") pod \"authorino-operator-76b478bd78-x9b85\" (UID: \"b3c4cd20-2303-48e9-9c7b-dd655487489e\") " pod="kuadrant-system/authorino-operator-76b478bd78-x9b85" Sep 08 16:57:22.294068 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:22.294007 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njx27\" (UniqueName: \"kubernetes.io/projected/b3c4cd20-2303-48e9-9c7b-dd655487489e-kube-api-access-njx27\") pod \"authorino-operator-76b478bd78-x9b85\" (UID: \"b3c4cd20-2303-48e9-9c7b-dd655487489e\") " pod="kuadrant-system/authorino-operator-76b478bd78-x9b85" Sep 08 16:57:22.313165 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:22.313144 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njx27\" (UniqueName: \"kubernetes.io/projected/b3c4cd20-2303-48e9-9c7b-dd655487489e-kube-api-access-njx27\") pod \"authorino-operator-76b478bd78-x9b85\" (UID: \"b3c4cd20-2303-48e9-9c7b-dd655487489e\") " pod="kuadrant-system/authorino-operator-76b478bd78-x9b85" Sep 08 16:57:22.458787 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:22.458756 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-x9b85" Sep 08 16:57:22.743870 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:22.743846 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-x9b85"] Sep 08 16:57:22.745779 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:57:22.745749 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c4cd20_2303_48e9_9c7b_dd655487489e.slice/crio-fe5e4ca166d2d2315242a28d01447cd55df193dfb6c69b89db072b619bf0648f WatchSource:0}: Error finding container fe5e4ca166d2d2315242a28d01447cd55df193dfb6c69b89db072b619bf0648f: Status 404 returned error can't find the container with id fe5e4ca166d2d2315242a28d01447cd55df193dfb6c69b89db072b619bf0648f Sep 08 16:57:23.387799 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:23.387755 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:23.693314 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:23.693280 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-x9b85" event={"ID":"b3c4cd20-2303-48e9-9c7b-dd655487489e","Type":"ContainerStarted","Data":"fe5e4ca166d2d2315242a28d01447cd55df193dfb6c69b89db072b619bf0648f"} Sep 08 16:57:24.337734 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:24.337696 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:24.392717 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:24.392684 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-c6fk2"] Sep 08 16:57:24.396161 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:24.396136 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-c6fk2" Sep 08 16:57:24.402004 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:24.401983 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 08 16:57:24.402566 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:24.402548 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-kx4t7\"" Sep 08 16:57:24.410285 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:24.410254 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8bp2\" (UniqueName: \"kubernetes.io/projected/1e6fef87-3f71-4022-9488-de1a691d34e3-kube-api-access-s8bp2\") pod \"dns-operator-controller-manager-7b84cd7c9c-c6fk2\" (UID: \"1e6fef87-3f71-4022-9488-de1a691d34e3\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-c6fk2" Sep 08 16:57:24.422411 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:24.422382 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-c6fk2"] Sep 08 16:57:24.511368 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:24.511333 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8bp2\" (UniqueName: \"kubernetes.io/projected/1e6fef87-3f71-4022-9488-de1a691d34e3-kube-api-access-s8bp2\") pod \"dns-operator-controller-manager-7b84cd7c9c-c6fk2\" (UID: \"1e6fef87-3f71-4022-9488-de1a691d34e3\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-c6fk2" Sep 08 16:57:24.559791 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:24.559757 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8bp2\" (UniqueName: \"kubernetes.io/projected/1e6fef87-3f71-4022-9488-de1a691d34e3-kube-api-access-s8bp2\") pod \"dns-operator-controller-manager-7b84cd7c9c-c6fk2\" (UID: \"1e6fef87-3f71-4022-9488-de1a691d34e3\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-c6fk2" Sep 08 16:57:24.708102 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:24.708065 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-c6fk2" Sep 08 16:57:24.821292 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:24.821262 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:24.999261 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:24.999182 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-c6fk2"] Sep 08 16:57:25.002132 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:57:25.002104 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6fef87_3f71_4022_9488_de1a691d34e3.slice/crio-724d2b985dd42b3426c15e66fcf0e837fe79f22b0b70e75d03aaeeaacaf0f5f2 WatchSource:0}: Error finding container 724d2b985dd42b3426c15e66fcf0e837fe79f22b0b70e75d03aaeeaacaf0f5f2: Status 404 returned error can't find the container with id 724d2b985dd42b3426c15e66fcf0e837fe79f22b0b70e75d03aaeeaacaf0f5f2 Sep 08 16:57:25.129183 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:25.129140 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:25.704986 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:25.704953 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-x9b85" event={"ID":"b3c4cd20-2303-48e9-9c7b-dd655487489e","Type":"ContainerStarted","Data":"2a6e1aceda6a1f7ae13d3785dcd6f1a748125b22a992727b751e98f27f889af3"} Sep 08 16:57:25.705412 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:25.705068 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-x9b85" Sep 08 16:57:25.706294 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:25.706269 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-c6fk2" event={"ID":"1e6fef87-3f71-4022-9488-de1a691d34e3","Type":"ContainerStarted","Data":"724d2b985dd42b3426c15e66fcf0e837fe79f22b0b70e75d03aaeeaacaf0f5f2"} Sep 08 16:57:25.792494 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:25.792433 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-x9b85" podStartSLOduration=1.7739892720000001 podStartE2EDuration="3.792401138s" totalImagesPullingTime="2.018411866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:57:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:57:25.789668451 +0000 UTC m=+408.526499564" watchObservedRunningTime="2026-09-08 16:57:25.792401138 +0000 UTC m=+408.529232258" Sep 08 16:57:25.986458 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:25.986342 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:26.199460 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:26.199400 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:27.718889 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:27.718856 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-c6fk2" event={"ID":"1e6fef87-3f71-4022-9488-de1a691d34e3","Type":"ContainerStarted","Data":"46eccb9dbeabf550e6a527ef92c97db87cd13e00dcb91f8eddec515c64529d2e"} Sep 08 16:57:27.719263 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:27.718997 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-c6fk2" Sep 08 16:57:27.759168 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:27.759113 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-c6fk2" podStartSLOduration=2.1091091889999998 podStartE2EDuration="3.7590984s" totalImagesPullingTime="1.649989211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:57:27.757633157 +0000 UTC m=+410.494464285" watchObservedRunningTime="2026-09-08 16:57:27.7590984 +0000 UTC m=+410.495929516" Sep 08 16:57:28.729439 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:28.729393 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn"] Sep 08 16:57:28.732870 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:28.732854 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" Sep 08 16:57:28.735997 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:28.735978 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jgzqv\"" Sep 08 16:57:28.823092 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:28.823069 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn"] Sep 08 16:57:28.847125 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:28.847102 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sth2\" (UniqueName: \"kubernetes.io/projected/eda8f79d-1026-411c-905a-a9d9a25a4032-kube-api-access-7sth2\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zcrjn\" (UID: \"eda8f79d-1026-411c-905a-a9d9a25a4032\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" Sep 08 16:57:28.847225 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:28.847146 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eda8f79d-1026-411c-905a-a9d9a25a4032-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zcrjn\" (UID: \"eda8f79d-1026-411c-905a-a9d9a25a4032\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" Sep 08 16:57:28.947556 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:28.947528 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eda8f79d-1026-411c-905a-a9d9a25a4032-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zcrjn\" (UID: \"eda8f79d-1026-411c-905a-a9d9a25a4032\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" Sep 08 16:57:28.947708 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:28.947586 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sth2\" (UniqueName: \"kubernetes.io/projected/eda8f79d-1026-411c-905a-a9d9a25a4032-kube-api-access-7sth2\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zcrjn\" (UID: \"eda8f79d-1026-411c-905a-a9d9a25a4032\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" Sep 08 16:57:28.947929 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:28.947907 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eda8f79d-1026-411c-905a-a9d9a25a4032-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zcrjn\" (UID: \"eda8f79d-1026-411c-905a-a9d9a25a4032\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" Sep 08 16:57:28.963617 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:28.963589 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sth2\" (UniqueName: \"kubernetes.io/projected/eda8f79d-1026-411c-905a-a9d9a25a4032-kube-api-access-7sth2\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zcrjn\" (UID: \"eda8f79d-1026-411c-905a-a9d9a25a4032\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" Sep 08 16:57:29.042980 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:29.042920 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" Sep 08 16:57:29.348877 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:29.348848 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn"] Sep 08 16:57:29.349448 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:57:29.349410 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda8f79d_1026_411c_905a_a9d9a25a4032.slice/crio-0dc2fe151dabe23456b6b61382a94cdfaa58ef045b4078fb673766cd6cac1363 WatchSource:0}: Error finding container 0dc2fe151dabe23456b6b61382a94cdfaa58ef045b4078fb673766cd6cac1363: Status 404 returned error can't find the container with id 0dc2fe151dabe23456b6b61382a94cdfaa58ef045b4078fb673766cd6cac1363 Sep 08 16:57:29.588117 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:29.588088 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:29.730201 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:29.730173 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" event={"ID":"eda8f79d-1026-411c-905a-a9d9a25a4032","Type":"ContainerStarted","Data":"0dc2fe151dabe23456b6b61382a94cdfaa58ef045b4078fb673766cd6cac1363"} Sep 08 16:57:29.843442 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:29.843394 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:32.689852 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:32.689822 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l" Sep 08 16:57:33.760387 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:33.760312 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" event={"ID":"eda8f79d-1026-411c-905a-a9d9a25a4032","Type":"ContainerStarted","Data":"f1b80b90a62f720bc107df11adc3107355902e2c8f0633c69ae286944611b5f5"} Sep 08 16:57:33.760387 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:33.760367 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" Sep 08 16:57:33.798363 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:33.798309 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" podStartSLOduration=1.703712932 podStartE2EDuration="5.798290052s" totalImagesPullingTime="4.09457712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:57:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:57:33.7982077 +0000 UTC m=+416.535038817" watchObservedRunningTime="2026-09-08 16:57:33.798290052 +0000 UTC m=+416.535121170" Sep 08 16:57:33.966355 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:33.966321 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:34.066686 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:34.066608 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:34.251847 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:34.251817 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:34.764957 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:34.764924 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:34.971396 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:34.971364 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:36.207934 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.207899 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:36.373627 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.373594 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-dm7hb"] Sep 08 16:57:36.378626 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.378604 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dm7hb" Sep 08 16:57:36.381559 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.381535 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 08 16:57:36.381670 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.381537 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9kmwg\"" Sep 08 16:57:36.385414 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.385397 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 08 16:57:36.387511 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.387481 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:36.398220 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.398198 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-dm7hb"] Sep 08 16:57:36.509716 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.509619 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/da60a5c3-4a54-4079-9218-842083016fa5-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-dm7hb\" (UID: \"da60a5c3-4a54-4079-9218-842083016fa5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dm7hb" Sep 08 16:57:36.509716 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.509666 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dds82\" (UniqueName: \"kubernetes.io/projected/da60a5c3-4a54-4079-9218-842083016fa5-kube-api-access-dds82\") pod \"kuadrant-console-plugin-fbdd4f445-dm7hb\" (UID: \"da60a5c3-4a54-4079-9218-842083016fa5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dm7hb" Sep 08 16:57:36.509905 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.509729 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/da60a5c3-4a54-4079-9218-842083016fa5-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-dm7hb\" (UID: \"da60a5c3-4a54-4079-9218-842083016fa5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dm7hb" Sep 08 16:57:36.611074 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.611030 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/da60a5c3-4a54-4079-9218-842083016fa5-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-dm7hb\" (UID: \"da60a5c3-4a54-4079-9218-842083016fa5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dm7hb" Sep 08 16:57:36.611074 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.611072 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dds82\" (UniqueName: \"kubernetes.io/projected/da60a5c3-4a54-4079-9218-842083016fa5-kube-api-access-dds82\") pod \"kuadrant-console-plugin-fbdd4f445-dm7hb\" (UID: \"da60a5c3-4a54-4079-9218-842083016fa5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dm7hb" Sep 08 16:57:36.611313 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.611119 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/da60a5c3-4a54-4079-9218-842083016fa5-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-dm7hb\" (UID: \"da60a5c3-4a54-4079-9218-842083016fa5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dm7hb" Sep 08 16:57:36.611714 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.611685 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/da60a5c3-4a54-4079-9218-842083016fa5-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-dm7hb\" (UID: \"da60a5c3-4a54-4079-9218-842083016fa5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dm7hb" Sep 08 16:57:36.613906 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.613882 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/da60a5c3-4a54-4079-9218-842083016fa5-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-dm7hb\" (UID: \"da60a5c3-4a54-4079-9218-842083016fa5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dm7hb" Sep 08 16:57:36.641722 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.641696 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b9cc75d5d-4766s"] Sep 08 16:57:36.645494 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.645475 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.646912 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.646890 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dds82\" (UniqueName: \"kubernetes.io/projected/da60a5c3-4a54-4079-9218-842083016fa5-kube-api-access-dds82\") pod \"kuadrant-console-plugin-fbdd4f445-dm7hb\" (UID: \"da60a5c3-4a54-4079-9218-842083016fa5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dm7hb" Sep 08 16:57:36.671955 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.671935 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:36.685698 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.685673 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b9cc75d5d-4766s"] Sep 08 16:57:36.688143 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.688121 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dm7hb" Sep 08 16:57:36.712479 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.712451 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/187a2cde-e262-4c91-94ab-dabffecb5b69-trusted-ca-bundle\") pod \"console-b9cc75d5d-4766s\" (UID: \"187a2cde-e262-4c91-94ab-dabffecb5b69\") " pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.712573 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.712489 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95mjd\" (UniqueName: \"kubernetes.io/projected/187a2cde-e262-4c91-94ab-dabffecb5b69-kube-api-access-95mjd\") pod \"console-b9cc75d5d-4766s\" (UID: \"187a2cde-e262-4c91-94ab-dabffecb5b69\") " pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.712573 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.712545 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/187a2cde-e262-4c91-94ab-dabffecb5b69-console-oauth-config\") pod \"console-b9cc75d5d-4766s\" (UID: \"187a2cde-e262-4c91-94ab-dabffecb5b69\") " pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.712573 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.712562 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/187a2cde-e262-4c91-94ab-dabffecb5b69-service-ca\") pod \"console-b9cc75d5d-4766s\" (UID: \"187a2cde-e262-4c91-94ab-dabffecb5b69\") " pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.712745 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.712642 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/187a2cde-e262-4c91-94ab-dabffecb5b69-oauth-serving-cert\") pod \"console-b9cc75d5d-4766s\" (UID: \"187a2cde-e262-4c91-94ab-dabffecb5b69\") " pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.712745 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.712698 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/187a2cde-e262-4c91-94ab-dabffecb5b69-console-serving-cert\") pod \"console-b9cc75d5d-4766s\" (UID: \"187a2cde-e262-4c91-94ab-dabffecb5b69\") " pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.712821 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.712793 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/187a2cde-e262-4c91-94ab-dabffecb5b69-console-config\") pod \"console-b9cc75d5d-4766s\" (UID: \"187a2cde-e262-4c91-94ab-dabffecb5b69\") " pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.714101 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.714080 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-x9b85" Sep 08 16:57:36.814010 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.813927 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/187a2cde-e262-4c91-94ab-dabffecb5b69-trusted-ca-bundle\") pod \"console-b9cc75d5d-4766s\" (UID: \"187a2cde-e262-4c91-94ab-dabffecb5b69\") " pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.814010 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.813969 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95mjd\" (UniqueName: \"kubernetes.io/projected/187a2cde-e262-4c91-94ab-dabffecb5b69-kube-api-access-95mjd\") pod \"console-b9cc75d5d-4766s\" (UID: \"187a2cde-e262-4c91-94ab-dabffecb5b69\") " pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.814238 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.814024 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/187a2cde-e262-4c91-94ab-dabffecb5b69-console-oauth-config\") pod \"console-b9cc75d5d-4766s\" (UID: \"187a2cde-e262-4c91-94ab-dabffecb5b69\") " pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.814238 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.814050 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/187a2cde-e262-4c91-94ab-dabffecb5b69-service-ca\") pod \"console-b9cc75d5d-4766s\" (UID: \"187a2cde-e262-4c91-94ab-dabffecb5b69\") " pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.814238 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.814087 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/187a2cde-e262-4c91-94ab-dabffecb5b69-oauth-serving-cert\") pod \"console-b9cc75d5d-4766s\" (UID: \"187a2cde-e262-4c91-94ab-dabffecb5b69\") " pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.814238 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.814131 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/187a2cde-e262-4c91-94ab-dabffecb5b69-console-serving-cert\") pod \"console-b9cc75d5d-4766s\" (UID: \"187a2cde-e262-4c91-94ab-dabffecb5b69\") " pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.814238 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.814165 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/187a2cde-e262-4c91-94ab-dabffecb5b69-console-config\") pod \"console-b9cc75d5d-4766s\" (UID: \"187a2cde-e262-4c91-94ab-dabffecb5b69\") " pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.814951 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.814917 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/187a2cde-e262-4c91-94ab-dabffecb5b69-console-config\") pod \"console-b9cc75d5d-4766s\" (UID: \"187a2cde-e262-4c91-94ab-dabffecb5b69\") " pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.815124 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.815100 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/187a2cde-e262-4c91-94ab-dabffecb5b69-trusted-ca-bundle\") pod \"console-b9cc75d5d-4766s\" (UID: \"187a2cde-e262-4c91-94ab-dabffecb5b69\") " pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.815194 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.815146 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/187a2cde-e262-4c91-94ab-dabffecb5b69-oauth-serving-cert\") pod \"console-b9cc75d5d-4766s\" (UID: \"187a2cde-e262-4c91-94ab-dabffecb5b69\") " pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.816199 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.816020 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/187a2cde-e262-4c91-94ab-dabffecb5b69-service-ca\") pod \"console-b9cc75d5d-4766s\" (UID: \"187a2cde-e262-4c91-94ab-dabffecb5b69\") " pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.816884 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.816760 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:36.817351 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.817317 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/187a2cde-e262-4c91-94ab-dabffecb5b69-console-oauth-config\") pod \"console-b9cc75d5d-4766s\" (UID: \"187a2cde-e262-4c91-94ab-dabffecb5b69\") " pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.817463 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.817348 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/187a2cde-e262-4c91-94ab-dabffecb5b69-console-serving-cert\") pod \"console-b9cc75d5d-4766s\" (UID: \"187a2cde-e262-4c91-94ab-dabffecb5b69\") " pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.830806 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.830785 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95mjd\" (UniqueName: \"kubernetes.io/projected/187a2cde-e262-4c91-94ab-dabffecb5b69-kube-api-access-95mjd\") pod \"console-b9cc75d5d-4766s\" (UID: \"187a2cde-e262-4c91-94ab-dabffecb5b69\") " pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.927299 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.927261 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:36.961999 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.961963 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:36.975680 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:36.975656 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-dm7hb"] Sep 08 16:57:36.977916 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:57:36.977876 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda60a5c3_4a54_4079_9218_842083016fa5.slice/crio-4315f26374a52c25372f7e330d7af4791d23ebb106bf8c41230b680c5375d2c6 WatchSource:0}: Error finding container 4315f26374a52c25372f7e330d7af4791d23ebb106bf8c41230b680c5375d2c6: Status 404 returned error can't find the container with id 4315f26374a52c25372f7e330d7af4791d23ebb106bf8c41230b680c5375d2c6 Sep 08 16:57:37.262121 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:37.262086 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b9cc75d5d-4766s"] Sep 08 16:57:37.266095 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:57:37.266073 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod187a2cde_e262_4c91_94ab_dabffecb5b69.slice/crio-15a864ed25401523f91f0c36c51b77962ccf99f67ac72a65226630cc8ae29ee2 WatchSource:0}: Error finding container 15a864ed25401523f91f0c36c51b77962ccf99f67ac72a65226630cc8ae29ee2: Status 404 returned error can't find the container with id 15a864ed25401523f91f0c36c51b77962ccf99f67ac72a65226630cc8ae29ee2 Sep 08 16:57:37.446774 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:37.446739 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:37.696962 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:37.696928 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:37.783287 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:37.783256 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b9cc75d5d-4766s" event={"ID":"187a2cde-e262-4c91-94ab-dabffecb5b69","Type":"ContainerStarted","Data":"f7b3eaeba018fd4c05050ccf94adf7515cda15a8598da708477dac2954938daa"} Sep 08 16:57:37.783466 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:37.783294 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b9cc75d5d-4766s" event={"ID":"187a2cde-e262-4c91-94ab-dabffecb5b69","Type":"ContainerStarted","Data":"15a864ed25401523f91f0c36c51b77962ccf99f67ac72a65226630cc8ae29ee2"} Sep 08 16:57:37.784576 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:37.784555 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dm7hb" event={"ID":"da60a5c3-4a54-4079-9218-842083016fa5","Type":"ContainerStarted","Data":"4315f26374a52c25372f7e330d7af4791d23ebb106bf8c41230b680c5375d2c6"} Sep 08 16:57:37.818982 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:37.818926 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b9cc75d5d-4766s" podStartSLOduration=1.818910936 podStartE2EDuration="1.818910936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:57:37.816454828 +0000 UTC m=+420.553285941" watchObservedRunningTime="2026-09-08 16:57:37.818910936 +0000 UTC m=+420.555742120" Sep 08 16:57:38.726842 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:38.726802 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-c6fk2" Sep 08 16:57:42.738308 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:42.738270 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:42.817863 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:42.817830 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dm7hb" event={"ID":"da60a5c3-4a54-4079-9218-842083016fa5","Type":"ContainerStarted","Data":"b7e9a6b9f70619d55779c686f2d843f7bdfd4d0f9dd02dce459ed8e3ba9797e4"} Sep 08 16:57:42.844150 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:42.844113 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:42.941468 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:42.941440 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:44.767316 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:44.767287 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" Sep 08 16:57:44.792468 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:44.792409 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dm7hb" podStartSLOduration=3.647760548 podStartE2EDuration="8.792395581s" totalImagesPullingTime="5.144635033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:57:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:57:42.840574661 +0000 UTC m=+425.577405776" watchObservedRunningTime="2026-09-08 16:57:44.792395581 +0000 UTC m=+427.529226744" Sep 08 16:57:46.843823 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:46.843784 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mdnr6"] Sep 08 16:57:46.846948 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:46.846930 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mdnr6" Sep 08 16:57:46.864307 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:46.864284 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mdnr6"] Sep 08 16:57:46.905609 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:46.905576 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v8mm\" (UniqueName: \"kubernetes.io/projected/164f074d-d1ee-410f-b481-82e3b37d7276-kube-api-access-7v8mm\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-mdnr6\" (UID: \"164f074d-d1ee-410f-b481-82e3b37d7276\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mdnr6" Sep 08 16:57:46.905752 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:46.905640 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/164f074d-d1ee-410f-b481-82e3b37d7276-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-mdnr6\" (UID: \"164f074d-d1ee-410f-b481-82e3b37d7276\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mdnr6" Sep 08 16:57:46.962592 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:46.962562 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:46.962716 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:46.962620 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:46.967207 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:46.967189 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:46.995819 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:46.995795 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mdnr6"] Sep 08 16:57:46.996016 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:57:46.995999 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-7v8mm], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mdnr6" podUID="164f074d-d1ee-410f-b481-82e3b37d7276" Sep 08 16:57:47.006008 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.005984 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v8mm\" (UniqueName: \"kubernetes.io/projected/164f074d-d1ee-410f-b481-82e3b37d7276-kube-api-access-7v8mm\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-mdnr6\" (UID: \"164f074d-d1ee-410f-b481-82e3b37d7276\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mdnr6" Sep 08 16:57:47.006107 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.006025 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/164f074d-d1ee-410f-b481-82e3b37d7276-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-mdnr6\" (UID: \"164f074d-d1ee-410f-b481-82e3b37d7276\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mdnr6" Sep 08 16:57:47.006354 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.006338 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/164f074d-d1ee-410f-b481-82e3b37d7276-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-mdnr6\" (UID: \"164f074d-d1ee-410f-b481-82e3b37d7276\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mdnr6" Sep 08 16:57:47.020146 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.020122 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v8mm\" (UniqueName: \"kubernetes.io/projected/164f074d-d1ee-410f-b481-82e3b37d7276-kube-api-access-7v8mm\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-mdnr6\" (UID: \"164f074d-d1ee-410f-b481-82e3b37d7276\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mdnr6" Sep 08 16:57:47.022781 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.022760 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn"] Sep 08 16:57:47.022977 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.022937 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" podUID="eda8f79d-1026-411c-905a-a9d9a25a4032" containerName="manager" containerID="cri-o://f1b80b90a62f720bc107df11adc3107355902e2c8f0633c69ae286944611b5f5" gracePeriod=2 Sep 08 16:57:47.024907 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:57:47.024861 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zcrjn\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="eda8f79d-1026-411c-905a-a9d9a25a4032" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" Sep 08 16:57:47.027449 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.027408 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn"] Sep 08 16:57:47.060455 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.060407 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mdnr6"] Sep 08 16:57:47.067479 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.067458 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t"] Sep 08 16:57:47.067820 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.067809 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eda8f79d-1026-411c-905a-a9d9a25a4032" containerName="manager" Sep 08 16:57:47.067866 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.067823 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eda8f79d-1026-411c-905a-a9d9a25a4032" containerName="manager" Sep 08 16:57:47.067904 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.067876 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="eda8f79d-1026-411c-905a-a9d9a25a4032" containerName="manager" Sep 08 16:57:47.070852 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.070837 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t" Sep 08 16:57:47.077509 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.077483 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mdnr6"] Sep 08 16:57:47.094738 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.094662 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l"] Sep 08 16:57:47.094894 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.094869 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l" podUID="c15d6121-3965-4732-9abc-3b1e2ca9b3c8" containerName="manager" containerID="cri-o://87ee02417f4f290254debb9c789def78497d2a90337eebd39891b2920da6a63e" gracePeriod=2 Sep 08 16:57:47.097146 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.097124 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj"] Sep 08 16:57:47.100676 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.100653 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj" Sep 08 16:57:47.106306 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:57:47.106279 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zcrjn\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="eda8f79d-1026-411c-905a-a9d9a25a4032" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" Sep 08 16:57:47.108358 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:57:47.108332 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zcrjn\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="eda8f79d-1026-411c-905a-a9d9a25a4032" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" Sep 08 16:57:47.108868 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.108850 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l"] Sep 08 16:57:47.111616 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:57:47.111589 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-s242l\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="c15d6121-3965-4732-9abc-3b1e2ca9b3c8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l" Sep 08 16:57:47.119223 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.119204 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj"] Sep 08 16:57:47.127648 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.127629 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fn9mx"] Sep 08 16:57:47.127938 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.127927 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c15d6121-3965-4732-9abc-3b1e2ca9b3c8" containerName="manager" Sep 08 16:57:47.127980 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.127940 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c15d6121-3965-4732-9abc-3b1e2ca9b3c8" containerName="manager" Sep 08 16:57:47.128021 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.128013 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c15d6121-3965-4732-9abc-3b1e2ca9b3c8" containerName="manager" Sep 08 16:57:47.131139 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.131119 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fn9mx" Sep 08 16:57:47.148304 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:57:47.148275 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-s242l\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="c15d6121-3965-4732-9abc-3b1e2ca9b3c8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l" Sep 08 16:57:47.152364 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.152342 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t"] Sep 08 16:57:47.154627 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.154602 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fn9mx"] Sep 08 16:57:47.176027 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:57:47.175998 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zcrjn\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="eda8f79d-1026-411c-905a-a9d9a25a4032" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" Sep 08 16:57:47.207231 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.207202 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fbb3ccf-a062-48e0-abc7-fe3d539b75dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v4kdj\" (UID: \"8fbb3ccf-a062-48e0-abc7-fe3d539b75dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj" Sep 08 16:57:47.207329 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.207248 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5ht5\" (UniqueName: \"kubernetes.io/projected/8fbb3ccf-a062-48e0-abc7-fe3d539b75dd-kube-api-access-l5ht5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v4kdj\" (UID: \"8fbb3ccf-a062-48e0-abc7-fe3d539b75dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj" Sep 08 16:57:47.207414 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.207390 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c46ht\" (UniqueName: \"kubernetes.io/projected/8b11eb00-7c73-4334-82cc-bc64fc094fa7-kube-api-access-c46ht\") pod \"limitador-operator-controller-manager-74495c69f8-fn9mx\" (UID: \"8b11eb00-7c73-4334-82cc-bc64fc094fa7\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fn9mx" Sep 08 16:57:47.207496 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.207477 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82b588c3-bcc8-4877-90cf-579a4e5eb5b3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cm29t\" (UID: \"82b588c3-bcc8-4877-90cf-579a4e5eb5b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t" Sep 08 16:57:47.207554 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.207538 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wdlf\" (UniqueName: \"kubernetes.io/projected/82b588c3-bcc8-4877-90cf-579a4e5eb5b3-kube-api-access-9wdlf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cm29t\" (UID: \"82b588c3-bcc8-4877-90cf-579a4e5eb5b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t" Sep 08 16:57:47.308887 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.308852 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fbb3ccf-a062-48e0-abc7-fe3d539b75dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v4kdj\" (UID: \"8fbb3ccf-a062-48e0-abc7-fe3d539b75dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj" Sep 08 16:57:47.308993 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.308903 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5ht5\" (UniqueName: \"kubernetes.io/projected/8fbb3ccf-a062-48e0-abc7-fe3d539b75dd-kube-api-access-l5ht5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v4kdj\" (UID: \"8fbb3ccf-a062-48e0-abc7-fe3d539b75dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj" Sep 08 16:57:47.308993 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.308964 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c46ht\" (UniqueName: \"kubernetes.io/projected/8b11eb00-7c73-4334-82cc-bc64fc094fa7-kube-api-access-c46ht\") pod \"limitador-operator-controller-manager-74495c69f8-fn9mx\" (UID: \"8b11eb00-7c73-4334-82cc-bc64fc094fa7\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fn9mx" Sep 08 16:57:47.309075 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.308993 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82b588c3-bcc8-4877-90cf-579a4e5eb5b3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cm29t\" (UID: \"82b588c3-bcc8-4877-90cf-579a4e5eb5b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t" Sep 08 16:57:47.309075 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.309028 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wdlf\" (UniqueName: \"kubernetes.io/projected/82b588c3-bcc8-4877-90cf-579a4e5eb5b3-kube-api-access-9wdlf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cm29t\" (UID: \"82b588c3-bcc8-4877-90cf-579a4e5eb5b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t" Sep 08 16:57:47.309264 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.309240 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fbb3ccf-a062-48e0-abc7-fe3d539b75dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v4kdj\" (UID: \"8fbb3ccf-a062-48e0-abc7-fe3d539b75dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj" Sep 08 16:57:47.309476 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.309456 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82b588c3-bcc8-4877-90cf-579a4e5eb5b3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cm29t\" (UID: \"82b588c3-bcc8-4877-90cf-579a4e5eb5b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t" Sep 08 16:57:47.325866 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.325843 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" Sep 08 16:57:47.331073 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:57:47.331039 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zcrjn\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="eda8f79d-1026-411c-905a-a9d9a25a4032" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" Sep 08 16:57:47.335340 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.335318 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5ht5\" (UniqueName: \"kubernetes.io/projected/8fbb3ccf-a062-48e0-abc7-fe3d539b75dd-kube-api-access-l5ht5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v4kdj\" (UID: \"8fbb3ccf-a062-48e0-abc7-fe3d539b75dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj" Sep 08 16:57:47.335738 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.335654 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wdlf\" (UniqueName: \"kubernetes.io/projected/82b588c3-bcc8-4877-90cf-579a4e5eb5b3-kube-api-access-9wdlf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cm29t\" (UID: \"82b588c3-bcc8-4877-90cf-579a4e5eb5b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t" Sep 08 16:57:47.339217 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:57:47.339180 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-s242l\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="c15d6121-3965-4732-9abc-3b1e2ca9b3c8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l" Sep 08 16:57:47.339472 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.339447 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c46ht\" (UniqueName: \"kubernetes.io/projected/8b11eb00-7c73-4334-82cc-bc64fc094fa7-kube-api-access-c46ht\") pod \"limitador-operator-controller-manager-74495c69f8-fn9mx\" (UID: \"8b11eb00-7c73-4334-82cc-bc64fc094fa7\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fn9mx" Sep 08 16:57:47.343344 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.343325 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l" Sep 08 16:57:47.345595 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:57:47.345535 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-s242l\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="c15d6121-3965-4732-9abc-3b1e2ca9b3c8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l" Sep 08 16:57:47.347550 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:57:47.347523 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zcrjn\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="eda8f79d-1026-411c-905a-a9d9a25a4032" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" Sep 08 16:57:47.409963 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.409941 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sth2\" (UniqueName: \"kubernetes.io/projected/eda8f79d-1026-411c-905a-a9d9a25a4032-kube-api-access-7sth2\") pod \"eda8f79d-1026-411c-905a-a9d9a25a4032\" (UID: \"eda8f79d-1026-411c-905a-a9d9a25a4032\") " Sep 08 16:57:47.410061 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.410015 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pblb4\" (UniqueName: \"kubernetes.io/projected/c15d6121-3965-4732-9abc-3b1e2ca9b3c8-kube-api-access-pblb4\") pod \"c15d6121-3965-4732-9abc-3b1e2ca9b3c8\" (UID: \"c15d6121-3965-4732-9abc-3b1e2ca9b3c8\") " Sep 08 16:57:47.410061 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.410046 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eda8f79d-1026-411c-905a-a9d9a25a4032-extensions-socket-volume\") pod \"eda8f79d-1026-411c-905a-a9d9a25a4032\" (UID: \"eda8f79d-1026-411c-905a-a9d9a25a4032\") " Sep 08 16:57:47.410487 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.410465 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda8f79d-1026-411c-905a-a9d9a25a4032-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "eda8f79d-1026-411c-905a-a9d9a25a4032" (UID: "eda8f79d-1026-411c-905a-a9d9a25a4032"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:57:47.411982 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.411962 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15d6121-3965-4732-9abc-3b1e2ca9b3c8-kube-api-access-pblb4" (OuterVolumeSpecName: "kube-api-access-pblb4") pod "c15d6121-3965-4732-9abc-3b1e2ca9b3c8" (UID: "c15d6121-3965-4732-9abc-3b1e2ca9b3c8"). InnerVolumeSpecName "kube-api-access-pblb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:57:47.412054 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.412000 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda8f79d-1026-411c-905a-a9d9a25a4032-kube-api-access-7sth2" (OuterVolumeSpecName: "kube-api-access-7sth2") pod "eda8f79d-1026-411c-905a-a9d9a25a4032" (UID: "eda8f79d-1026-411c-905a-a9d9a25a4032"). InnerVolumeSpecName "kube-api-access-7sth2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:57:47.469506 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.469486 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t" Sep 08 16:57:47.478334 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.478314 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj" Sep 08 16:57:47.484259 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.484241 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fn9mx" Sep 08 16:57:47.511835 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.511764 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sth2\" (UniqueName: \"kubernetes.io/projected/eda8f79d-1026-411c-905a-a9d9a25a4032-kube-api-access-7sth2\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:57:47.511835 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.511797 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pblb4\" (UniqueName: \"kubernetes.io/projected/c15d6121-3965-4732-9abc-3b1e2ca9b3c8-kube-api-access-pblb4\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:57:47.511835 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.511821 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eda8f79d-1026-411c-905a-a9d9a25a4032-extensions-socket-volume\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:57:47.761450 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.761073 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c15d6121-3965-4732-9abc-3b1e2ca9b3c8" path="/var/lib/kubelet/pods/c15d6121-3965-4732-9abc-3b1e2ca9b3c8/volumes" Sep 08 16:57:47.764202 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.761671 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eda8f79d-1026-411c-905a-a9d9a25a4032" path="/var/lib/kubelet/pods/eda8f79d-1026-411c-905a-a9d9a25a4032/volumes" Sep 08 16:57:47.765617 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:57:47.765583 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zcrjn\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="eda8f79d-1026-411c-905a-a9d9a25a4032" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" Sep 08 16:57:47.768893 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:57:47.768862 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-s242l\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="c15d6121-3965-4732-9abc-3b1e2ca9b3c8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l" Sep 08 16:57:47.777063 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.776947 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj"] Sep 08 16:57:47.782932 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:57:47.782907 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbb3ccf_a062_48e0_abc7_fe3d539b75dd.slice/crio-f6a621333305b91bf0d69bc2b5b7467aaa318f7525212986406ef0c42644dfd8 WatchSource:0}: Error finding container f6a621333305b91bf0d69bc2b5b7467aaa318f7525212986406ef0c42644dfd8: Status 404 returned error can't find the container with id f6a621333305b91bf0d69bc2b5b7467aaa318f7525212986406ef0c42644dfd8 Sep 08 16:57:47.783031 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.782944 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t"] Sep 08 16:57:47.783342 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:57:47.783322 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b588c3_bcc8_4877_90cf_579a4e5eb5b3.slice/crio-4f15a420dee0482b2ff6a482f248225229f1833609731adc04bbdd6f05b71077 WatchSource:0}: Error finding container 4f15a420dee0482b2ff6a482f248225229f1833609731adc04bbdd6f05b71077: Status 404 returned error can't find the container with id 4f15a420dee0482b2ff6a482f248225229f1833609731adc04bbdd6f05b71077 Sep 08 16:57:47.790198 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.790176 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fn9mx"] Sep 08 16:57:47.790887 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:57:47.790867 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b11eb00_7c73_4334_82cc_bc64fc094fa7.slice/crio-6a3b36024fae09ca4450fe12117a7ff646dcb0964349db8bd721f702e80b7359 WatchSource:0}: Error finding container 6a3b36024fae09ca4450fe12117a7ff646dcb0964349db8bd721f702e80b7359: Status 404 returned error can't find the container with id 6a3b36024fae09ca4450fe12117a7ff646dcb0964349db8bd721f702e80b7359 Sep 08 16:57:47.848994 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.848959 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t" event={"ID":"82b588c3-bcc8-4877-90cf-579a4e5eb5b3","Type":"ContainerStarted","Data":"4f15a420dee0482b2ff6a482f248225229f1833609731adc04bbdd6f05b71077"} Sep 08 16:57:47.851678 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.851649 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj" event={"ID":"8fbb3ccf-a062-48e0-abc7-fe3d539b75dd","Type":"ContainerStarted","Data":"f6a621333305b91bf0d69bc2b5b7467aaa318f7525212986406ef0c42644dfd8"} Sep 08 16:57:47.853296 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.853275 2565 generic.go:356] "Generic (PLEG): container finished" podID="c15d6121-3965-4732-9abc-3b1e2ca9b3c8" containerID="87ee02417f4f290254debb9c789def78497d2a90337eebd39891b2920da6a63e" exitCode=0 Sep 08 16:57:47.853385 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.853344 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s242l" Sep 08 16:57:47.853385 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.853368 2565 scope.go:117] "RemoveContainer" containerID="87ee02417f4f290254debb9c789def78497d2a90337eebd39891b2920da6a63e" Sep 08 16:57:47.855164 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.855137 2565 generic.go:356] "Generic (PLEG): container finished" podID="eda8f79d-1026-411c-905a-a9d9a25a4032" containerID="f1b80b90a62f720bc107df11adc3107355902e2c8f0633c69ae286944611b5f5" exitCode=0 Sep 08 16:57:47.855340 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.855322 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zcrjn" Sep 08 16:57:47.857576 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.857546 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mdnr6" Sep 08 16:57:47.857674 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.857575 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fn9mx" event={"ID":"8b11eb00-7c73-4334-82cc-bc64fc094fa7","Type":"ContainerStarted","Data":"6a3b36024fae09ca4450fe12117a7ff646dcb0964349db8bd721f702e80b7359"} Sep 08 16:57:47.858349 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.858329 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fn9mx" Sep 08 16:57:47.863076 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.863044 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b9cc75d5d-4766s" Sep 08 16:57:47.866014 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.865950 2565 scope.go:117] "RemoveContainer" containerID="87ee02417f4f290254debb9c789def78497d2a90337eebd39891b2920da6a63e" Sep 08 16:57:47.866282 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:57:47.866264 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ee02417f4f290254debb9c789def78497d2a90337eebd39891b2920da6a63e\": container with ID starting with 87ee02417f4f290254debb9c789def78497d2a90337eebd39891b2920da6a63e not found: ID does not exist" containerID="87ee02417f4f290254debb9c789def78497d2a90337eebd39891b2920da6a63e" Sep 08 16:57:47.866478 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.866289 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ee02417f4f290254debb9c789def78497d2a90337eebd39891b2920da6a63e"} err="failed to get container status \"87ee02417f4f290254debb9c789def78497d2a90337eebd39891b2920da6a63e\": rpc error: code = NotFound desc = could not find container \"87ee02417f4f290254debb9c789def78497d2a90337eebd39891b2920da6a63e\": container with ID starting with 87ee02417f4f290254debb9c789def78497d2a90337eebd39891b2920da6a63e not found: ID does not exist" Sep 08 16:57:47.866478 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.866305 2565 scope.go:117] "RemoveContainer" containerID="f1b80b90a62f720bc107df11adc3107355902e2c8f0633c69ae286944611b5f5" Sep 08 16:57:47.868418 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.868399 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mdnr6" Sep 08 16:57:47.874583 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.874564 2565 scope.go:117] "RemoveContainer" containerID="f1b80b90a62f720bc107df11adc3107355902e2c8f0633c69ae286944611b5f5" Sep 08 16:57:47.874830 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:57:47.874813 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1b80b90a62f720bc107df11adc3107355902e2c8f0633c69ae286944611b5f5\": container with ID starting with f1b80b90a62f720bc107df11adc3107355902e2c8f0633c69ae286944611b5f5 not found: ID does not exist" containerID="f1b80b90a62f720bc107df11adc3107355902e2c8f0633c69ae286944611b5f5" Sep 08 16:57:47.874883 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.874836 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b80b90a62f720bc107df11adc3107355902e2c8f0633c69ae286944611b5f5"} err="failed to get container status \"f1b80b90a62f720bc107df11adc3107355902e2c8f0633c69ae286944611b5f5\": rpc error: code = NotFound desc = could not find container \"f1b80b90a62f720bc107df11adc3107355902e2c8f0633c69ae286944611b5f5\": container with ID starting with f1b80b90a62f720bc107df11adc3107355902e2c8f0633c69ae286944611b5f5 not found: ID does not exist" Sep 08 16:57:47.893655 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:57:47.893629 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-mdnr6\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="164f074d-d1ee-410f-b481-82e3b37d7276" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mdnr6" Sep 08 16:57:47.897006 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:47.896971 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fn9mx" podStartSLOduration=0.896958761 podStartE2EDuration="896.958761ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:57:47.891077221 +0000 UTC m=+430.627908337" watchObservedRunningTime="2026-09-08 16:57:47.896958761 +0000 UTC m=+430.633789882" Sep 08 16:57:48.014676 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:48.014601 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v8mm\" (UniqueName: \"kubernetes.io/projected/164f074d-d1ee-410f-b481-82e3b37d7276-kube-api-access-7v8mm\") pod \"164f074d-d1ee-410f-b481-82e3b37d7276\" (UID: \"164f074d-d1ee-410f-b481-82e3b37d7276\") " Sep 08 16:57:48.014676 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:48.014674 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/164f074d-d1ee-410f-b481-82e3b37d7276-extensions-socket-volume\") pod \"164f074d-d1ee-410f-b481-82e3b37d7276\" (UID: \"164f074d-d1ee-410f-b481-82e3b37d7276\") " Sep 08 16:57:48.015027 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:48.014945 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/164f074d-d1ee-410f-b481-82e3b37d7276-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "164f074d-d1ee-410f-b481-82e3b37d7276" (UID: "164f074d-d1ee-410f-b481-82e3b37d7276"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:57:48.015125 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:48.015096 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/164f074d-d1ee-410f-b481-82e3b37d7276-extensions-socket-volume\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:57:48.016763 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:48.016732 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/164f074d-d1ee-410f-b481-82e3b37d7276-kube-api-access-7v8mm" (OuterVolumeSpecName: "kube-api-access-7v8mm") pod "164f074d-d1ee-410f-b481-82e3b37d7276" (UID: "164f074d-d1ee-410f-b481-82e3b37d7276"). InnerVolumeSpecName "kube-api-access-7v8mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:57:48.062056 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:48.062022 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-89b7c5f89-c7qk9"] Sep 08 16:57:48.115802 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:48.115768 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v8mm\" (UniqueName: \"kubernetes.io/projected/164f074d-d1ee-410f-b481-82e3b37d7276-kube-api-access-7v8mm\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:57:48.175717 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:57:48.175675 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-mdnr6\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="164f074d-d1ee-410f-b481-82e3b37d7276" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mdnr6" Sep 08 16:57:48.748246 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:48.748208 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:48.860818 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:48.860784 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:48.866225 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:48.866186 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fn9mx" event={"ID":"8b11eb00-7c73-4334-82cc-bc64fc094fa7","Type":"ContainerStarted","Data":"9778f96612473606773ec7f26834a3c33fb476704f93f7fc65d183eea9d75cd4"} Sep 08 16:57:48.867931 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:48.867902 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t" event={"ID":"82b588c3-bcc8-4877-90cf-579a4e5eb5b3","Type":"ContainerStarted","Data":"44765351b0e6362bb1e7e1d9c3cfec99ae05200ee2ce3ad5555f55f1a28d4043"} Sep 08 16:57:48.868064 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:48.868047 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t" Sep 08 16:57:48.869679 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:48.869650 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj" event={"ID":"8fbb3ccf-a062-48e0-abc7-fe3d539b75dd","Type":"ContainerStarted","Data":"3add475a965a79a382cb22ab38247a8693389d81135e8299df1449b5383fccec"} Sep 08 16:57:48.869791 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:48.869701 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj" Sep 08 16:57:48.870634 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:48.870614 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mdnr6" Sep 08 16:57:48.930445 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:48.930382 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t" podStartSLOduration=1.930362416 podStartE2EDuration="1.930362416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:57:48.929395066 +0000 UTC m=+431.666226181" watchObservedRunningTime="2026-09-08 16:57:48.930362416 +0000 UTC m=+431.667193532" Sep 08 16:57:48.931365 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:57:48.931342 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-mdnr6\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="164f074d-d1ee-410f-b481-82e3b37d7276" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mdnr6" Sep 08 16:57:48.933220 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:57:48.933197 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-mdnr6\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="164f074d-d1ee-410f-b481-82e3b37d7276" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-mdnr6" Sep 08 16:57:49.570180 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:49.570145 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:49.732287 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:49.732253 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:49.761063 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:49.761020 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="164f074d-d1ee-410f-b481-82e3b37d7276" path="/var/lib/kubelet/pods/164f074d-d1ee-410f-b481-82e3b37d7276/volumes" Sep 08 16:57:53.243202 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:53.243167 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:53.370115 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:53.370070 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:53.524935 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:53.524846 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:57.003945 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:57.003907 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:57.117489 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:57.117457 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:57:59.878195 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:59.878164 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fn9mx" Sep 08 16:57:59.878657 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:59.878244 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj" Sep 08 16:57:59.878657 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:59.878290 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t" Sep 08 16:57:59.932033 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:57:59.931971 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj" podStartSLOduration=12.93195714 podStartE2EDuration="12.93195714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:57:48.967303122 +0000 UTC m=+431.704134240" watchObservedRunningTime="2026-09-08 16:57:59.93195714 +0000 UTC m=+442.668788256" Sep 08 16:58:00.215300 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:00.215268 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj"] Sep 08 16:58:00.215522 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:00.215499 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj" podUID="8fbb3ccf-a062-48e0-abc7-fe3d539b75dd" containerName="manager" containerID="cri-o://3add475a965a79a382cb22ab38247a8693389d81135e8299df1449b5383fccec" gracePeriod=10 Sep 08 16:58:00.461455 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:00.461412 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj" Sep 08 16:58:00.611574 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:00.611547 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5ht5\" (UniqueName: \"kubernetes.io/projected/8fbb3ccf-a062-48e0-abc7-fe3d539b75dd-kube-api-access-l5ht5\") pod \"8fbb3ccf-a062-48e0-abc7-fe3d539b75dd\" (UID: \"8fbb3ccf-a062-48e0-abc7-fe3d539b75dd\") " Sep 08 16:58:00.611689 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:00.611599 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fbb3ccf-a062-48e0-abc7-fe3d539b75dd-extensions-socket-volume\") pod \"8fbb3ccf-a062-48e0-abc7-fe3d539b75dd\" (UID: \"8fbb3ccf-a062-48e0-abc7-fe3d539b75dd\") " Sep 08 16:58:00.612038 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:00.612011 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fbb3ccf-a062-48e0-abc7-fe3d539b75dd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8fbb3ccf-a062-48e0-abc7-fe3d539b75dd" (UID: "8fbb3ccf-a062-48e0-abc7-fe3d539b75dd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:58:00.613651 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:00.613625 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbb3ccf-a062-48e0-abc7-fe3d539b75dd-kube-api-access-l5ht5" (OuterVolumeSpecName: "kube-api-access-l5ht5") pod "8fbb3ccf-a062-48e0-abc7-fe3d539b75dd" (UID: "8fbb3ccf-a062-48e0-abc7-fe3d539b75dd"). InnerVolumeSpecName "kube-api-access-l5ht5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:58:00.712258 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:00.712226 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5ht5\" (UniqueName: \"kubernetes.io/projected/8fbb3ccf-a062-48e0-abc7-fe3d539b75dd-kube-api-access-l5ht5\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:58:00.712258 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:00.712252 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fbb3ccf-a062-48e0-abc7-fe3d539b75dd-extensions-socket-volume\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:58:00.947062 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:00.947029 2565 generic.go:356] "Generic (PLEG): container finished" podID="8fbb3ccf-a062-48e0-abc7-fe3d539b75dd" containerID="3add475a965a79a382cb22ab38247a8693389d81135e8299df1449b5383fccec" exitCode=0 Sep 08 16:58:00.947461 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:00.947081 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj" Sep 08 16:58:00.947461 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:00.947117 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj" event={"ID":"8fbb3ccf-a062-48e0-abc7-fe3d539b75dd","Type":"ContainerDied","Data":"3add475a965a79a382cb22ab38247a8693389d81135e8299df1449b5383fccec"} Sep 08 16:58:00.947461 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:00.947192 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj" event={"ID":"8fbb3ccf-a062-48e0-abc7-fe3d539b75dd","Type":"ContainerDied","Data":"f6a621333305b91bf0d69bc2b5b7467aaa318f7525212986406ef0c42644dfd8"} Sep 08 16:58:00.947461 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:00.947220 2565 scope.go:117] "RemoveContainer" containerID="3add475a965a79a382cb22ab38247a8693389d81135e8299df1449b5383fccec" Sep 08 16:58:00.956550 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:00.956519 2565 scope.go:117] "RemoveContainer" containerID="3add475a965a79a382cb22ab38247a8693389d81135e8299df1449b5383fccec" Sep 08 16:58:00.956813 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:58:00.956794 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3add475a965a79a382cb22ab38247a8693389d81135e8299df1449b5383fccec\": container with ID starting with 3add475a965a79a382cb22ab38247a8693389d81135e8299df1449b5383fccec not found: ID does not exist" containerID="3add475a965a79a382cb22ab38247a8693389d81135e8299df1449b5383fccec" Sep 08 16:58:00.956878 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:00.956821 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3add475a965a79a382cb22ab38247a8693389d81135e8299df1449b5383fccec"} err="failed to get container status \"3add475a965a79a382cb22ab38247a8693389d81135e8299df1449b5383fccec\": rpc error: code = NotFound desc = could not find container \"3add475a965a79a382cb22ab38247a8693389d81135e8299df1449b5383fccec\": container with ID starting with 3add475a965a79a382cb22ab38247a8693389d81135e8299df1449b5383fccec not found: ID does not exist" Sep 08 16:58:00.991720 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:00.991694 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj"] Sep 08 16:58:00.997851 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:00.997832 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4kdj"] Sep 08 16:58:01.759607 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:01.759578 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fbb3ccf-a062-48e0-abc7-fe3d539b75dd" path="/var/lib/kubelet/pods/8fbb3ccf-a062-48e0-abc7-fe3d539b75dd/volumes" Sep 08 16:58:03.147769 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:03.147736 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:58:03.294105 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:03.294076 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:58:03.392047 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:03.392018 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:58:13.086367 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.086319 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-89b7c5f89-c7qk9" podUID="3ae831f9-ffb0-4484-9b05-d888780464d1" containerName="console" containerID="cri-o://d8cd4455900a9c068462f35bf6efeb80b507211798b52802ec6188e656832578" gracePeriod=15 Sep 08 16:58:13.332118 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.332093 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:58:13.508487 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.508456 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-trusted-ca-bundle\") pod \"3ae831f9-ffb0-4484-9b05-d888780464d1\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " Sep 08 16:58:13.508642 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.508517 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-service-ca\") pod \"3ae831f9-ffb0-4484-9b05-d888780464d1\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " Sep 08 16:58:13.508642 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.508556 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-console-config\") pod \"3ae831f9-ffb0-4484-9b05-d888780464d1\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " Sep 08 16:58:13.508642 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.508579 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-oauth-serving-cert\") pod \"3ae831f9-ffb0-4484-9b05-d888780464d1\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " Sep 08 16:58:13.508642 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.508604 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg2gp\" (UniqueName: \"kubernetes.io/projected/3ae831f9-ffb0-4484-9b05-d888780464d1-kube-api-access-jg2gp\") pod \"3ae831f9-ffb0-4484-9b05-d888780464d1\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " Sep 08 16:58:13.508865 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.508754 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ae831f9-ffb0-4484-9b05-d888780464d1-console-serving-cert\") pod \"3ae831f9-ffb0-4484-9b05-d888780464d1\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " Sep 08 16:58:13.508865 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.508802 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ae831f9-ffb0-4484-9b05-d888780464d1-console-oauth-config\") pod \"3ae831f9-ffb0-4484-9b05-d888780464d1\" (UID: \"3ae831f9-ffb0-4484-9b05-d888780464d1\") " Sep 08 16:58:13.508968 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.508948 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3ae831f9-ffb0-4484-9b05-d888780464d1" (UID: "3ae831f9-ffb0-4484-9b05-d888780464d1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:58:13.509029 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.509004 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-service-ca" (OuterVolumeSpecName: "service-ca") pod "3ae831f9-ffb0-4484-9b05-d888780464d1" (UID: "3ae831f9-ffb0-4484-9b05-d888780464d1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:58:13.509029 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.509011 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3ae831f9-ffb0-4484-9b05-d888780464d1" (UID: "3ae831f9-ffb0-4484-9b05-d888780464d1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:58:13.509120 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.508996 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-console-config" (OuterVolumeSpecName: "console-config") pod "3ae831f9-ffb0-4484-9b05-d888780464d1" (UID: "3ae831f9-ffb0-4484-9b05-d888780464d1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:58:13.509154 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.509117 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-console-config\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:58:13.509154 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.509131 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-oauth-serving-cert\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:58:13.509154 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.509140 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-trusted-ca-bundle\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:58:13.509154 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.509148 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae831f9-ffb0-4484-9b05-d888780464d1-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:58:13.510943 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.510914 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ae831f9-ffb0-4484-9b05-d888780464d1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3ae831f9-ffb0-4484-9b05-d888780464d1" (UID: "3ae831f9-ffb0-4484-9b05-d888780464d1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:58:13.510943 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.510919 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae831f9-ffb0-4484-9b05-d888780464d1-kube-api-access-jg2gp" (OuterVolumeSpecName: "kube-api-access-jg2gp") pod "3ae831f9-ffb0-4484-9b05-d888780464d1" (UID: "3ae831f9-ffb0-4484-9b05-d888780464d1"). InnerVolumeSpecName "kube-api-access-jg2gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:58:13.510943 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.510933 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ae831f9-ffb0-4484-9b05-d888780464d1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3ae831f9-ffb0-4484-9b05-d888780464d1" (UID: "3ae831f9-ffb0-4484-9b05-d888780464d1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:58:13.610165 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.610129 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ae831f9-ffb0-4484-9b05-d888780464d1-console-serving-cert\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:58:13.610165 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.610163 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ae831f9-ffb0-4484-9b05-d888780464d1-console-oauth-config\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:58:13.610165 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:13.610173 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg2gp\" (UniqueName: \"kubernetes.io/projected/3ae831f9-ffb0-4484-9b05-d888780464d1-kube-api-access-jg2gp\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:58:14.020548 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:14.020476 2565 generic.go:356] "Generic (PLEG): container finished" podID="3ae831f9-ffb0-4484-9b05-d888780464d1" containerID="d8cd4455900a9c068462f35bf6efeb80b507211798b52802ec6188e656832578" exitCode=0 Sep 08 16:58:14.020548 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:14.020519 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-89b7c5f89-c7qk9" Sep 08 16:58:14.020548 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:14.020533 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-89b7c5f89-c7qk9" event={"ID":"3ae831f9-ffb0-4484-9b05-d888780464d1","Type":"ContainerDied","Data":"d8cd4455900a9c068462f35bf6efeb80b507211798b52802ec6188e656832578"} Sep 08 16:58:14.020742 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:14.020585 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-89b7c5f89-c7qk9" event={"ID":"3ae831f9-ffb0-4484-9b05-d888780464d1","Type":"ContainerDied","Data":"fa2743446b03915d6ae3ebc9566cd8c7c1ef6b7946ca634955a7a87efb458fa2"} Sep 08 16:58:14.020742 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:14.020606 2565 scope.go:117] "RemoveContainer" containerID="d8cd4455900a9c068462f35bf6efeb80b507211798b52802ec6188e656832578" Sep 08 16:58:14.029704 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:14.029688 2565 scope.go:117] "RemoveContainer" containerID="d8cd4455900a9c068462f35bf6efeb80b507211798b52802ec6188e656832578" Sep 08 16:58:14.029960 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:58:14.029940 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8cd4455900a9c068462f35bf6efeb80b507211798b52802ec6188e656832578\": container with ID starting with d8cd4455900a9c068462f35bf6efeb80b507211798b52802ec6188e656832578 not found: ID does not exist" containerID="d8cd4455900a9c068462f35bf6efeb80b507211798b52802ec6188e656832578" Sep 08 16:58:14.030026 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:14.029974 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8cd4455900a9c068462f35bf6efeb80b507211798b52802ec6188e656832578"} err="failed to get container status \"d8cd4455900a9c068462f35bf6efeb80b507211798b52802ec6188e656832578\": rpc error: code = NotFound desc = could not find container \"d8cd4455900a9c068462f35bf6efeb80b507211798b52802ec6188e656832578\": container with ID starting with d8cd4455900a9c068462f35bf6efeb80b507211798b52802ec6188e656832578 not found: ID does not exist" Sep 08 16:58:14.044208 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:14.044180 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-89b7c5f89-c7qk9"] Sep 08 16:58:14.049761 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:14.049739 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-89b7c5f89-c7qk9"] Sep 08 16:58:15.760086 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:15.760055 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ae831f9-ffb0-4484-9b05-d888780464d1" path="/var/lib/kubelet/pods/3ae831f9-ffb0-4484-9b05-d888780464d1/volumes" Sep 08 16:58:16.839905 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:16.839872 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:58:16.936127 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:16.936097 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:58:17.362768 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:17.362696 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:58:19.417338 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:19.417298 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:58:19.422744 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:19.422719 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xwz6w"] Sep 08 16:58:19.423122 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:19.423108 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ae831f9-ffb0-4484-9b05-d888780464d1" containerName="console" Sep 08 16:58:19.423177 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:19.423125 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ae831f9-ffb0-4484-9b05-d888780464d1" containerName="console" Sep 08 16:58:19.423177 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:19.423140 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fbb3ccf-a062-48e0-abc7-fe3d539b75dd" containerName="manager" Sep 08 16:58:19.423177 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:19.423145 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fbb3ccf-a062-48e0-abc7-fe3d539b75dd" containerName="manager" Sep 08 16:58:19.423271 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:19.423216 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ae831f9-ffb0-4484-9b05-d888780464d1" containerName="console" Sep 08 16:58:19.423271 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:19.423224 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fbb3ccf-a062-48e0-abc7-fe3d539b75dd" containerName="manager" Sep 08 16:58:19.445999 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:19.445979 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xwz6w"] Sep 08 16:58:19.446118 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:19.446063 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xwz6w" Sep 08 16:58:19.450456 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:19.450437 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 08 16:58:19.522000 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:19.521967 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xwz6w"] Sep 08 16:58:19.529648 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:19.529614 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:58:19.553722 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:19.553695 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn4l5\" (UniqueName: \"kubernetes.io/projected/23a2ad39-3f14-409c-b440-f054541bf9b3-kube-api-access-dn4l5\") pod \"limitador-limitador-8588749977-xwz6w\" (UID: \"23a2ad39-3f14-409c-b440-f054541bf9b3\") " pod="kuadrant-system/limitador-limitador-8588749977-xwz6w" Sep 08 16:58:19.553834 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:19.553751 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/23a2ad39-3f14-409c-b440-f054541bf9b3-config-file\") pod \"limitador-limitador-8588749977-xwz6w\" (UID: \"23a2ad39-3f14-409c-b440-f054541bf9b3\") " pod="kuadrant-system/limitador-limitador-8588749977-xwz6w" Sep 08 16:58:19.627316 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:19.627285 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:58:19.654313 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:19.654281 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn4l5\" (UniqueName: \"kubernetes.io/projected/23a2ad39-3f14-409c-b440-f054541bf9b3-kube-api-access-dn4l5\") pod \"limitador-limitador-8588749977-xwz6w\" (UID: \"23a2ad39-3f14-409c-b440-f054541bf9b3\") " pod="kuadrant-system/limitador-limitador-8588749977-xwz6w" Sep 08 16:58:19.654456 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:19.654344 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/23a2ad39-3f14-409c-b440-f054541bf9b3-config-file\") pod \"limitador-limitador-8588749977-xwz6w\" (UID: \"23a2ad39-3f14-409c-b440-f054541bf9b3\") " pod="kuadrant-system/limitador-limitador-8588749977-xwz6w" Sep 08 16:58:19.654988 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:19.654969 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/23a2ad39-3f14-409c-b440-f054541bf9b3-config-file\") pod \"limitador-limitador-8588749977-xwz6w\" (UID: \"23a2ad39-3f14-409c-b440-f054541bf9b3\") " pod="kuadrant-system/limitador-limitador-8588749977-xwz6w" Sep 08 16:58:19.668000 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:19.667945 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn4l5\" (UniqueName: \"kubernetes.io/projected/23a2ad39-3f14-409c-b440-f054541bf9b3-kube-api-access-dn4l5\") pod \"limitador-limitador-8588749977-xwz6w\" (UID: \"23a2ad39-3f14-409c-b440-f054541bf9b3\") " pod="kuadrant-system/limitador-limitador-8588749977-xwz6w" Sep 08 16:58:19.756535 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:19.756504 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xwz6w" Sep 08 16:58:20.032092 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:20.032049 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xwz6w"] Sep 08 16:58:20.033922 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:58:20.033893 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a2ad39_3f14_409c_b440_f054541bf9b3.slice/crio-92c2700817c03dcb8edec5ee601b43b34b1e8d999728e732862552558f3e6252 WatchSource:0}: Error finding container 92c2700817c03dcb8edec5ee601b43b34b1e8d999728e732862552558f3e6252: Status 404 returned error can't find the container with id 92c2700817c03dcb8edec5ee601b43b34b1e8d999728e732862552558f3e6252 Sep 08 16:58:20.054397 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:20.054361 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xwz6w" event={"ID":"23a2ad39-3f14-409c-b440-f054541bf9b3","Type":"ContainerStarted","Data":"92c2700817c03dcb8edec5ee601b43b34b1e8d999728e732862552558f3e6252"} Sep 08 16:58:20.308551 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:20.308473 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n2zs2"] Sep 08 16:58:20.313547 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:20.313525 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-n2zs2" Sep 08 16:58:20.316341 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:20.316324 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zzdrs\"" Sep 08 16:58:20.330756 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:20.330730 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n2zs2"] Sep 08 16:58:20.332981 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:20.332961 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-qmhp5"] Sep 08 16:58:20.336394 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:20.336374 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qmhp5" Sep 08 16:58:20.344865 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:20.344843 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qmhp5"] Sep 08 16:58:20.360665 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:20.360643 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jrjh\" (UniqueName: \"kubernetes.io/projected/c5a04aa5-610c-4361-9722-dc11bce684e8-kube-api-access-9jrjh\") pod \"authorino-7bcfd8f9cd-n2zs2\" (UID: \"c5a04aa5-610c-4361-9722-dc11bce684e8\") " pod="kuadrant-system/authorino-7bcfd8f9cd-n2zs2" Sep 08 16:58:20.462038 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:20.462004 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jrjh\" (UniqueName: \"kubernetes.io/projected/c5a04aa5-610c-4361-9722-dc11bce684e8-kube-api-access-9jrjh\") pod \"authorino-7bcfd8f9cd-n2zs2\" (UID: \"c5a04aa5-610c-4361-9722-dc11bce684e8\") " pod="kuadrant-system/authorino-7bcfd8f9cd-n2zs2" Sep 08 16:58:20.462418 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:20.462075 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz6bx\" (UniqueName: \"kubernetes.io/projected/e28aa243-4554-4ffc-b885-b1352a4b4439-kube-api-access-wz6bx\") pod \"authorino-59c974bd48-qmhp5\" (UID: \"e28aa243-4554-4ffc-b885-b1352a4b4439\") " pod="kuadrant-system/authorino-59c974bd48-qmhp5" Sep 08 16:58:20.473890 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:20.473866 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jrjh\" (UniqueName: \"kubernetes.io/projected/c5a04aa5-610c-4361-9722-dc11bce684e8-kube-api-access-9jrjh\") pod \"authorino-7bcfd8f9cd-n2zs2\" (UID: \"c5a04aa5-610c-4361-9722-dc11bce684e8\") " pod="kuadrant-system/authorino-7bcfd8f9cd-n2zs2" Sep 08 16:58:20.563262 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:20.563197 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz6bx\" (UniqueName: \"kubernetes.io/projected/e28aa243-4554-4ffc-b885-b1352a4b4439-kube-api-access-wz6bx\") pod \"authorino-59c974bd48-qmhp5\" (UID: \"e28aa243-4554-4ffc-b885-b1352a4b4439\") " pod="kuadrant-system/authorino-59c974bd48-qmhp5" Sep 08 16:58:20.572848 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:20.572815 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz6bx\" (UniqueName: \"kubernetes.io/projected/e28aa243-4554-4ffc-b885-b1352a4b4439-kube-api-access-wz6bx\") pod \"authorino-59c974bd48-qmhp5\" (UID: \"e28aa243-4554-4ffc-b885-b1352a4b4439\") " pod="kuadrant-system/authorino-59c974bd48-qmhp5" Sep 08 16:58:20.622501 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:20.622474 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-n2zs2" Sep 08 16:58:20.645406 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:20.645381 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qmhp5" Sep 08 16:58:20.913382 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:20.913354 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n2zs2"] Sep 08 16:58:20.949589 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:20.949562 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qmhp5"] Sep 08 16:58:20.985032 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:58:20.985001 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5a04aa5_610c_4361_9722_dc11bce684e8.slice/crio-6aa6da57a8c4c89a9e5248c391aef8344cd614967b98af93a94bd3c418a5becf WatchSource:0}: Error finding container 6aa6da57a8c4c89a9e5248c391aef8344cd614967b98af93a94bd3c418a5becf: Status 404 returned error can't find the container with id 6aa6da57a8c4c89a9e5248c391aef8344cd614967b98af93a94bd3c418a5becf Sep 08 16:58:20.985828 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:58:20.985721 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode28aa243_4554_4ffc_b885_b1352a4b4439.slice/crio-8db2c1f7745cc82573e19a78c9ebfe49b512b4d70bfd56d03cd5fea853b8aad7 WatchSource:0}: Error finding container 8db2c1f7745cc82573e19a78c9ebfe49b512b4d70bfd56d03cd5fea853b8aad7: Status 404 returned error can't find the container with id 8db2c1f7745cc82573e19a78c9ebfe49b512b4d70bfd56d03cd5fea853b8aad7 Sep 08 16:58:21.063310 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:21.063274 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qmhp5" event={"ID":"e28aa243-4554-4ffc-b885-b1352a4b4439","Type":"ContainerStarted","Data":"8db2c1f7745cc82573e19a78c9ebfe49b512b4d70bfd56d03cd5fea853b8aad7"} Sep 08 16:58:21.064558 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:21.064533 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-n2zs2" event={"ID":"c5a04aa5-610c-4361-9722-dc11bce684e8","Type":"ContainerStarted","Data":"6aa6da57a8c4c89a9e5248c391aef8344cd614967b98af93a94bd3c418a5becf"} Sep 08 16:58:21.232138 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:21.232103 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:58:22.075563 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:22.075516 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xwz6w" event={"ID":"23a2ad39-3f14-409c-b440-f054541bf9b3","Type":"ContainerStarted","Data":"9cff5cf21e90ba9f53239c68d0327c8ea340b35f819a9e73395bc40d47c77e6a"} Sep 08 16:58:22.076028 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:22.075713 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-xwz6w" Sep 08 16:58:22.093647 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:22.093618 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:58:22.102066 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:22.101980 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-xwz6w" podStartSLOduration=2.095424134 podStartE2EDuration="3.101965201s" totalImagesPullingTime="1.006541067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:58:22.098893723 +0000 UTC m=+464.835724839" watchObservedRunningTime="2026-09-08 16:58:22.101965201 +0000 UTC m=+464.838796316" Sep 08 16:58:23.083076 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:23.082979 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qmhp5" event={"ID":"e28aa243-4554-4ffc-b885-b1352a4b4439","Type":"ContainerStarted","Data":"bfe75d885b6da89ce53a56d44618df78f45d134f6c48adb541954a764282c567"} Sep 08 16:58:23.084681 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:23.084655 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-n2zs2" event={"ID":"c5a04aa5-610c-4361-9722-dc11bce684e8","Type":"ContainerStarted","Data":"7a77fd6576b23ebea7675cbedbbd3c13bcb7e74c7f1e36a971b05e800c142ca3"} Sep 08 16:58:23.136309 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:23.136240 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-n2zs2" podStartSLOduration=1.310856569 podStartE2EDuration="3.136218318s" totalImagesPullingTime="1.825361749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:58:23.134945199 +0000 UTC m=+465.871776315" watchObservedRunningTime="2026-09-08 16:58:23.136218318 +0000 UTC m=+465.873049435" Sep 08 16:58:23.136497 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:23.136384 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-qmhp5" podStartSLOduration=1.315790825 podStartE2EDuration="3.136370431s" totalImagesPullingTime="1.820579606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:58:23.109530187 +0000 UTC m=+465.846361304" watchObservedRunningTime="2026-09-08 16:58:23.136370431 +0000 UTC m=+465.873201548" Sep 08 16:58:23.186255 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:23.186211 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n2zs2"] Sep 08 16:58:25.096019 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:25.095978 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-n2zs2" podUID="c5a04aa5-610c-4361-9722-dc11bce684e8" containerName="authorino" containerID="cri-o://7a77fd6576b23ebea7675cbedbbd3c13bcb7e74c7f1e36a971b05e800c142ca3" gracePeriod=30 Sep 08 16:58:25.340608 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:25.340585 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-n2zs2" Sep 08 16:58:25.404669 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:25.404591 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jrjh\" (UniqueName: \"kubernetes.io/projected/c5a04aa5-610c-4361-9722-dc11bce684e8-kube-api-access-9jrjh\") pod \"c5a04aa5-610c-4361-9722-dc11bce684e8\" (UID: \"c5a04aa5-610c-4361-9722-dc11bce684e8\") " Sep 08 16:58:25.406691 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:25.406670 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a04aa5-610c-4361-9722-dc11bce684e8-kube-api-access-9jrjh" (OuterVolumeSpecName: "kube-api-access-9jrjh") pod "c5a04aa5-610c-4361-9722-dc11bce684e8" (UID: "c5a04aa5-610c-4361-9722-dc11bce684e8"). InnerVolumeSpecName "kube-api-access-9jrjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:58:25.505348 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:25.505314 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jrjh\" (UniqueName: \"kubernetes.io/projected/c5a04aa5-610c-4361-9722-dc11bce684e8-kube-api-access-9jrjh\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:58:26.108919 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:26.108831 2565 generic.go:356] "Generic (PLEG): container finished" podID="c5a04aa5-610c-4361-9722-dc11bce684e8" containerID="7a77fd6576b23ebea7675cbedbbd3c13bcb7e74c7f1e36a971b05e800c142ca3" exitCode=0 Sep 08 16:58:26.108919 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:26.108868 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-n2zs2" event={"ID":"c5a04aa5-610c-4361-9722-dc11bce684e8","Type":"ContainerDied","Data":"7a77fd6576b23ebea7675cbedbbd3c13bcb7e74c7f1e36a971b05e800c142ca3"} Sep 08 16:58:26.108919 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:26.108909 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-n2zs2" event={"ID":"c5a04aa5-610c-4361-9722-dc11bce684e8","Type":"ContainerDied","Data":"6aa6da57a8c4c89a9e5248c391aef8344cd614967b98af93a94bd3c418a5becf"} Sep 08 16:58:26.109521 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:26.108926 2565 scope.go:117] "RemoveContainer" containerID="7a77fd6576b23ebea7675cbedbbd3c13bcb7e74c7f1e36a971b05e800c142ca3" Sep 08 16:58:26.109521 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:26.108877 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-n2zs2" Sep 08 16:58:26.120232 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:26.120209 2565 scope.go:117] "RemoveContainer" containerID="7a77fd6576b23ebea7675cbedbbd3c13bcb7e74c7f1e36a971b05e800c142ca3" Sep 08 16:58:26.120672 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:58:26.120650 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a77fd6576b23ebea7675cbedbbd3c13bcb7e74c7f1e36a971b05e800c142ca3\": container with ID starting with 7a77fd6576b23ebea7675cbedbbd3c13bcb7e74c7f1e36a971b05e800c142ca3 not found: ID does not exist" containerID="7a77fd6576b23ebea7675cbedbbd3c13bcb7e74c7f1e36a971b05e800c142ca3" Sep 08 16:58:26.120747 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:26.120686 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a77fd6576b23ebea7675cbedbbd3c13bcb7e74c7f1e36a971b05e800c142ca3"} err="failed to get container status \"7a77fd6576b23ebea7675cbedbbd3c13bcb7e74c7f1e36a971b05e800c142ca3\": rpc error: code = NotFound desc = could not find container \"7a77fd6576b23ebea7675cbedbbd3c13bcb7e74c7f1e36a971b05e800c142ca3\": container with ID starting with 7a77fd6576b23ebea7675cbedbbd3c13bcb7e74c7f1e36a971b05e800c142ca3 not found: ID does not exist" Sep 08 16:58:26.125490 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:26.125469 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n2zs2"] Sep 08 16:58:26.130508 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:26.130487 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n2zs2"] Sep 08 16:58:27.759739 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:27.759705 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5a04aa5-610c-4361-9722-dc11bce684e8" path="/var/lib/kubelet/pods/c5a04aa5-610c-4361-9722-dc11bce684e8/volumes" Sep 08 16:58:33.085595 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:33.085562 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-xwz6w" Sep 08 16:58:35.454440 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:35.454388 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xwz6w"] Sep 08 16:58:35.454840 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:35.454601 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-xwz6w" podUID="23a2ad39-3f14-409c-b440-f054541bf9b3" containerName="limitador" containerID="cri-o://9cff5cf21e90ba9f53239c68d0327c8ea340b35f819a9e73395bc40d47c77e6a" gracePeriod=30 Sep 08 16:58:36.003255 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:36.003227 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xwz6w" Sep 08 16:58:36.090177 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:36.090109 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dn4l5\" (UniqueName: \"kubernetes.io/projected/23a2ad39-3f14-409c-b440-f054541bf9b3-kube-api-access-dn4l5\") pod \"23a2ad39-3f14-409c-b440-f054541bf9b3\" (UID: \"23a2ad39-3f14-409c-b440-f054541bf9b3\") " Sep 08 16:58:36.090301 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:36.090180 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/23a2ad39-3f14-409c-b440-f054541bf9b3-config-file\") pod \"23a2ad39-3f14-409c-b440-f054541bf9b3\" (UID: \"23a2ad39-3f14-409c-b440-f054541bf9b3\") " Sep 08 16:58:36.090557 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:36.090533 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a2ad39-3f14-409c-b440-f054541bf9b3-config-file" (OuterVolumeSpecName: "config-file") pod "23a2ad39-3f14-409c-b440-f054541bf9b3" (UID: "23a2ad39-3f14-409c-b440-f054541bf9b3"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:58:36.092123 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:36.092104 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a2ad39-3f14-409c-b440-f054541bf9b3-kube-api-access-dn4l5" (OuterVolumeSpecName: "kube-api-access-dn4l5") pod "23a2ad39-3f14-409c-b440-f054541bf9b3" (UID: "23a2ad39-3f14-409c-b440-f054541bf9b3"). InnerVolumeSpecName "kube-api-access-dn4l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:58:36.166201 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:36.166166 2565 generic.go:356] "Generic (PLEG): container finished" podID="23a2ad39-3f14-409c-b440-f054541bf9b3" containerID="9cff5cf21e90ba9f53239c68d0327c8ea340b35f819a9e73395bc40d47c77e6a" exitCode=0 Sep 08 16:58:36.166344 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:36.166208 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xwz6w" Sep 08 16:58:36.166344 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:36.166212 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xwz6w" event={"ID":"23a2ad39-3f14-409c-b440-f054541bf9b3","Type":"ContainerDied","Data":"9cff5cf21e90ba9f53239c68d0327c8ea340b35f819a9e73395bc40d47c77e6a"} Sep 08 16:58:36.166344 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:36.166268 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xwz6w" event={"ID":"23a2ad39-3f14-409c-b440-f054541bf9b3","Type":"ContainerDied","Data":"92c2700817c03dcb8edec5ee601b43b34b1e8d999728e732862552558f3e6252"} Sep 08 16:58:36.166344 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:36.166292 2565 scope.go:117] "RemoveContainer" containerID="9cff5cf21e90ba9f53239c68d0327c8ea340b35f819a9e73395bc40d47c77e6a" Sep 08 16:58:36.176357 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:36.176341 2565 scope.go:117] "RemoveContainer" containerID="9cff5cf21e90ba9f53239c68d0327c8ea340b35f819a9e73395bc40d47c77e6a" Sep 08 16:58:36.176613 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:58:36.176595 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cff5cf21e90ba9f53239c68d0327c8ea340b35f819a9e73395bc40d47c77e6a\": container with ID starting with 9cff5cf21e90ba9f53239c68d0327c8ea340b35f819a9e73395bc40d47c77e6a not found: ID does not exist" containerID="9cff5cf21e90ba9f53239c68d0327c8ea340b35f819a9e73395bc40d47c77e6a" Sep 08 16:58:36.176677 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:36.176620 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cff5cf21e90ba9f53239c68d0327c8ea340b35f819a9e73395bc40d47c77e6a"} err="failed to get container status \"9cff5cf21e90ba9f53239c68d0327c8ea340b35f819a9e73395bc40d47c77e6a\": rpc error: code = NotFound desc = could not find container \"9cff5cf21e90ba9f53239c68d0327c8ea340b35f819a9e73395bc40d47c77e6a\": container with ID starting with 9cff5cf21e90ba9f53239c68d0327c8ea340b35f819a9e73395bc40d47c77e6a not found: ID does not exist" Sep 08 16:58:36.191505 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:36.191484 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn4l5\" (UniqueName: \"kubernetes.io/projected/23a2ad39-3f14-409c-b440-f054541bf9b3-kube-api-access-dn4l5\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:58:36.191584 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:36.191505 2565 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/23a2ad39-3f14-409c-b440-f054541bf9b3-config-file\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:58:36.203442 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:36.203403 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xwz6w"] Sep 08 16:58:36.210044 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:36.210012 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xwz6w"] Sep 08 16:58:37.759667 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:37.759635 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23a2ad39-3f14-409c-b440-f054541bf9b3" path="/var/lib/kubelet/pods/23a2ad39-3f14-409c-b440-f054541bf9b3/volumes" Sep 08 16:58:40.762047 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:40.762011 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:58:40.865026 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:40.864986 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:58:41.011871 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:41.011838 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-657vn"] Sep 08 16:58:41.012335 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:41.012264 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a04aa5-610c-4361-9722-dc11bce684e8" containerName="authorino" Sep 08 16:58:41.012335 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:41.012282 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a04aa5-610c-4361-9722-dc11bce684e8" containerName="authorino" Sep 08 16:58:41.012335 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:41.012306 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23a2ad39-3f14-409c-b440-f054541bf9b3" containerName="limitador" Sep 08 16:58:41.012335 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:41.012313 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="23a2ad39-3f14-409c-b440-f054541bf9b3" containerName="limitador" Sep 08 16:58:41.012675 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:41.012372 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5a04aa5-610c-4361-9722-dc11bce684e8" containerName="authorino" Sep 08 16:58:41.012675 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:41.012384 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="23a2ad39-3f14-409c-b440-f054541bf9b3" containerName="limitador" Sep 08 16:58:41.017055 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:41.017038 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-657vn" Sep 08 16:58:41.019595 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:41.019573 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 08 16:58:41.020361 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:41.020342 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 16:58:41.020475 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:41.020347 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-v6zts\"" Sep 08 16:58:41.020631 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:41.020615 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 08 16:58:41.027079 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:41.027052 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-657vn"] Sep 08 16:58:41.049833 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:41.049806 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:58:41.131678 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:41.131639 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fccab80a-e7c5-4811-ba95-294ac01ab453-data\") pod \"postgres-69fb4594cf-657vn\" (UID: \"fccab80a-e7c5-4811-ba95-294ac01ab453\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-657vn" Sep 08 16:58:41.131826 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:41.131707 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlvbw\" (UniqueName: \"kubernetes.io/projected/fccab80a-e7c5-4811-ba95-294ac01ab453-kube-api-access-wlvbw\") pod \"postgres-69fb4594cf-657vn\" (UID: \"fccab80a-e7c5-4811-ba95-294ac01ab453\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-657vn" Sep 08 16:58:41.232603 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:41.232567 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlvbw\" (UniqueName: \"kubernetes.io/projected/fccab80a-e7c5-4811-ba95-294ac01ab453-kube-api-access-wlvbw\") pod \"postgres-69fb4594cf-657vn\" (UID: \"fccab80a-e7c5-4811-ba95-294ac01ab453\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-657vn" Sep 08 16:58:41.232764 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:41.232656 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fccab80a-e7c5-4811-ba95-294ac01ab453-data\") pod \"postgres-69fb4594cf-657vn\" (UID: \"fccab80a-e7c5-4811-ba95-294ac01ab453\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-657vn" Sep 08 16:58:41.233013 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:41.232991 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fccab80a-e7c5-4811-ba95-294ac01ab453-data\") pod \"postgres-69fb4594cf-657vn\" (UID: \"fccab80a-e7c5-4811-ba95-294ac01ab453\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-657vn" Sep 08 16:58:41.246641 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:41.246618 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlvbw\" (UniqueName: \"kubernetes.io/projected/fccab80a-e7c5-4811-ba95-294ac01ab453-kube-api-access-wlvbw\") pod \"postgres-69fb4594cf-657vn\" (UID: \"fccab80a-e7c5-4811-ba95-294ac01ab453\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-657vn" Sep 08 16:58:41.328481 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:41.328383 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-657vn" Sep 08 16:58:41.600612 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:41.600583 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-657vn"] Sep 08 16:58:41.602532 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:58:41.602502 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfccab80a_e7c5_4811_ba95_294ac01ab453.slice/crio-3edcd62d3104772140e5c041ef6cb00dbb8b7c9d1932ea04a39308becb77734a WatchSource:0}: Error finding container 3edcd62d3104772140e5c041ef6cb00dbb8b7c9d1932ea04a39308becb77734a: Status 404 returned error can't find the container with id 3edcd62d3104772140e5c041ef6cb00dbb8b7c9d1932ea04a39308becb77734a Sep 08 16:58:42.206251 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:42.206206 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-657vn" event={"ID":"fccab80a-e7c5-4811-ba95-294ac01ab453","Type":"ContainerStarted","Data":"3edcd62d3104772140e5c041ef6cb00dbb8b7c9d1932ea04a39308becb77734a"} Sep 08 16:58:44.220162 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:44.220132 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-657vn" event={"ID":"fccab80a-e7c5-4811-ba95-294ac01ab453","Type":"ContainerStarted","Data":"b35a9c90723006bce5d6dd7927e5d30047cf327eb1e60335c41d4e3ba0286a2f"} Sep 08 16:58:44.220548 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:44.220251 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-657vn" Sep 08 16:58:44.240065 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:44.240019 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-657vn" podStartSLOduration=1.962813461 podStartE2EDuration="4.240006739s" totalImagesPullingTime="2.277193278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:58:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:58:44.237689084 +0000 UTC m=+486.974520202" watchObservedRunningTime="2026-09-08 16:58:44.240006739 +0000 UTC m=+486.976837854" Sep 08 16:58:50.255239 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:50.255207 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-657vn" Sep 08 16:58:50.586416 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:50.586327 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:58:50.753796 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:50.753763 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:58:50.803810 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:50.803779 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-4j64x"] Sep 08 16:58:50.808517 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:50.808499 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4j64x" Sep 08 16:58:50.817393 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:50.817366 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-4j64x"] Sep 08 16:58:50.921002 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:50.920944 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8lm7\" (UniqueName: \"kubernetes.io/projected/4acad7ef-98c3-4228-8e98-8729bff5c1e0-kube-api-access-z8lm7\") pod \"authorino-557f876f69-4j64x\" (UID: \"4acad7ef-98c3-4228-8e98-8729bff5c1e0\") " pod="kuadrant-system/authorino-557f876f69-4j64x" Sep 08 16:58:50.965879 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:50.965855 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-4j64x"] Sep 08 16:58:50.966064 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:58:50.966047 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-z8lm7], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-4j64x" podUID="4acad7ef-98c3-4228-8e98-8729bff5c1e0" Sep 08 16:58:51.008057 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.008033 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-qtxbj"] Sep 08 16:58:51.011380 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.011365 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-qtxbj" Sep 08 16:58:51.016366 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.016349 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 08 16:58:51.021650 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.021628 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8lm7\" (UniqueName: \"kubernetes.io/projected/4acad7ef-98c3-4228-8e98-8729bff5c1e0-kube-api-access-z8lm7\") pod \"authorino-557f876f69-4j64x\" (UID: \"4acad7ef-98c3-4228-8e98-8729bff5c1e0\") " pod="kuadrant-system/authorino-557f876f69-4j64x" Sep 08 16:58:51.033610 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.033589 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-qtxbj"] Sep 08 16:58:51.042930 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.042909 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8lm7\" (UniqueName: \"kubernetes.io/projected/4acad7ef-98c3-4228-8e98-8729bff5c1e0-kube-api-access-z8lm7\") pod \"authorino-557f876f69-4j64x\" (UID: \"4acad7ef-98c3-4228-8e98-8729bff5c1e0\") " pod="kuadrant-system/authorino-557f876f69-4j64x" Sep 08 16:58:51.111515 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.111486 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-qtxbj"] Sep 08 16:58:51.111713 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:58:51.111696 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-rfp5t tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-qtxbj" podUID="794b4bc4-ffa9-4432-bbc8-251d2cb55e48" Sep 08 16:58:51.122458 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.122437 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfp5t\" (UniqueName: \"kubernetes.io/projected/794b4bc4-ffa9-4432-bbc8-251d2cb55e48-kube-api-access-rfp5t\") pod \"authorino-6d4dfc649f-qtxbj\" (UID: \"794b4bc4-ffa9-4432-bbc8-251d2cb55e48\") " pod="kuadrant-system/authorino-6d4dfc649f-qtxbj" Sep 08 16:58:51.122558 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.122471 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/794b4bc4-ffa9-4432-bbc8-251d2cb55e48-tls-cert\") pod \"authorino-6d4dfc649f-qtxbj\" (UID: \"794b4bc4-ffa9-4432-bbc8-251d2cb55e48\") " pod="kuadrant-system/authorino-6d4dfc649f-qtxbj" Sep 08 16:58:51.169867 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.169838 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7d85785848-4jsdp"] Sep 08 16:58:51.173328 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.173280 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d85785848-4jsdp" Sep 08 16:58:51.183951 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.183929 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d85785848-4jsdp"] Sep 08 16:58:51.223058 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.223032 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1e2192da-d2a1-42c3-bbe3-31b1e061e306-tls-cert\") pod \"authorino-7d85785848-4jsdp\" (UID: \"1e2192da-d2a1-42c3-bbe3-31b1e061e306\") " pod="kuadrant-system/authorino-7d85785848-4jsdp" Sep 08 16:58:51.223154 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.223069 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22q42\" (UniqueName: \"kubernetes.io/projected/1e2192da-d2a1-42c3-bbe3-31b1e061e306-kube-api-access-22q42\") pod \"authorino-7d85785848-4jsdp\" (UID: \"1e2192da-d2a1-42c3-bbe3-31b1e061e306\") " pod="kuadrant-system/authorino-7d85785848-4jsdp" Sep 08 16:58:51.223210 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.223152 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfp5t\" (UniqueName: \"kubernetes.io/projected/794b4bc4-ffa9-4432-bbc8-251d2cb55e48-kube-api-access-rfp5t\") pod \"authorino-6d4dfc649f-qtxbj\" (UID: \"794b4bc4-ffa9-4432-bbc8-251d2cb55e48\") " pod="kuadrant-system/authorino-6d4dfc649f-qtxbj" Sep 08 16:58:51.223210 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.223196 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/794b4bc4-ffa9-4432-bbc8-251d2cb55e48-tls-cert\") pod \"authorino-6d4dfc649f-qtxbj\" (UID: \"794b4bc4-ffa9-4432-bbc8-251d2cb55e48\") " pod="kuadrant-system/authorino-6d4dfc649f-qtxbj" Sep 08 16:58:51.225477 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.225458 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/794b4bc4-ffa9-4432-bbc8-251d2cb55e48-tls-cert\") pod \"authorino-6d4dfc649f-qtxbj\" (UID: \"794b4bc4-ffa9-4432-bbc8-251d2cb55e48\") " pod="kuadrant-system/authorino-6d4dfc649f-qtxbj" Sep 08 16:58:51.234238 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.234216 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfp5t\" (UniqueName: \"kubernetes.io/projected/794b4bc4-ffa9-4432-bbc8-251d2cb55e48-kube-api-access-rfp5t\") pod \"authorino-6d4dfc649f-qtxbj\" (UID: \"794b4bc4-ffa9-4432-bbc8-251d2cb55e48\") " pod="kuadrant-system/authorino-6d4dfc649f-qtxbj" Sep 08 16:58:51.258909 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.258887 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4j64x" Sep 08 16:58:51.259195 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.258887 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-qtxbj" Sep 08 16:58:51.265873 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.265843 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4j64x" Sep 08 16:58:51.268913 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.268897 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-qtxbj" Sep 08 16:58:51.324371 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.324348 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfp5t\" (UniqueName: \"kubernetes.io/projected/794b4bc4-ffa9-4432-bbc8-251d2cb55e48-kube-api-access-rfp5t\") pod \"794b4bc4-ffa9-4432-bbc8-251d2cb55e48\" (UID: \"794b4bc4-ffa9-4432-bbc8-251d2cb55e48\") " Sep 08 16:58:51.324513 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.324389 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/794b4bc4-ffa9-4432-bbc8-251d2cb55e48-tls-cert\") pod \"794b4bc4-ffa9-4432-bbc8-251d2cb55e48\" (UID: \"794b4bc4-ffa9-4432-bbc8-251d2cb55e48\") " Sep 08 16:58:51.324513 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.324415 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8lm7\" (UniqueName: \"kubernetes.io/projected/4acad7ef-98c3-4228-8e98-8729bff5c1e0-kube-api-access-z8lm7\") pod \"4acad7ef-98c3-4228-8e98-8729bff5c1e0\" (UID: \"4acad7ef-98c3-4228-8e98-8729bff5c1e0\") " Sep 08 16:58:51.324640 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.324626 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1e2192da-d2a1-42c3-bbe3-31b1e061e306-tls-cert\") pod \"authorino-7d85785848-4jsdp\" (UID: \"1e2192da-d2a1-42c3-bbe3-31b1e061e306\") " pod="kuadrant-system/authorino-7d85785848-4jsdp" Sep 08 16:58:51.324740 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.324719 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22q42\" (UniqueName: \"kubernetes.io/projected/1e2192da-d2a1-42c3-bbe3-31b1e061e306-kube-api-access-22q42\") pod \"authorino-7d85785848-4jsdp\" (UID: \"1e2192da-d2a1-42c3-bbe3-31b1e061e306\") " pod="kuadrant-system/authorino-7d85785848-4jsdp" Sep 08 16:58:51.326481 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.326450 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4acad7ef-98c3-4228-8e98-8729bff5c1e0-kube-api-access-z8lm7" (OuterVolumeSpecName: "kube-api-access-z8lm7") pod "4acad7ef-98c3-4228-8e98-8729bff5c1e0" (UID: "4acad7ef-98c3-4228-8e98-8729bff5c1e0"). InnerVolumeSpecName "kube-api-access-z8lm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:58:51.326911 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.326883 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794b4bc4-ffa9-4432-bbc8-251d2cb55e48-kube-api-access-rfp5t" (OuterVolumeSpecName: "kube-api-access-rfp5t") pod "794b4bc4-ffa9-4432-bbc8-251d2cb55e48" (UID: "794b4bc4-ffa9-4432-bbc8-251d2cb55e48"). InnerVolumeSpecName "kube-api-access-rfp5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:58:51.326911 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.326886 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/794b4bc4-ffa9-4432-bbc8-251d2cb55e48-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "794b4bc4-ffa9-4432-bbc8-251d2cb55e48" (UID: "794b4bc4-ffa9-4432-bbc8-251d2cb55e48"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:58:51.327031 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.326985 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1e2192da-d2a1-42c3-bbe3-31b1e061e306-tls-cert\") pod \"authorino-7d85785848-4jsdp\" (UID: \"1e2192da-d2a1-42c3-bbe3-31b1e061e306\") " pod="kuadrant-system/authorino-7d85785848-4jsdp" Sep 08 16:58:51.333558 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.333536 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22q42\" (UniqueName: \"kubernetes.io/projected/1e2192da-d2a1-42c3-bbe3-31b1e061e306-kube-api-access-22q42\") pod \"authorino-7d85785848-4jsdp\" (UID: \"1e2192da-d2a1-42c3-bbe3-31b1e061e306\") " pod="kuadrant-system/authorino-7d85785848-4jsdp" Sep 08 16:58:51.425464 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.425390 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8lm7\" (UniqueName: \"kubernetes.io/projected/4acad7ef-98c3-4228-8e98-8729bff5c1e0-kube-api-access-z8lm7\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:58:51.425464 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.425413 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfp5t\" (UniqueName: \"kubernetes.io/projected/794b4bc4-ffa9-4432-bbc8-251d2cb55e48-kube-api-access-rfp5t\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:58:51.425464 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.425441 2565 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/794b4bc4-ffa9-4432-bbc8-251d2cb55e48-tls-cert\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:58:51.482785 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.482757 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d85785848-4jsdp" Sep 08 16:58:51.762326 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:51.762282 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d85785848-4jsdp"] Sep 08 16:58:51.764640 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:58:51.764612 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e2192da_d2a1_42c3_bbe3_31b1e061e306.slice/crio-2f1858e06ccd375529eb1e56c8ddbd5aae2ce9975a070c0f7d2a846a9021415f WatchSource:0}: Error finding container 2f1858e06ccd375529eb1e56c8ddbd5aae2ce9975a070c0f7d2a846a9021415f: Status 404 returned error can't find the container with id 2f1858e06ccd375529eb1e56c8ddbd5aae2ce9975a070c0f7d2a846a9021415f Sep 08 16:58:52.266430 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:52.266395 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d85785848-4jsdp" event={"ID":"1e2192da-d2a1-42c3-bbe3-31b1e061e306","Type":"ContainerStarted","Data":"2f1858e06ccd375529eb1e56c8ddbd5aae2ce9975a070c0f7d2a846a9021415f"} Sep 08 16:58:52.266789 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:52.266523 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4j64x" Sep 08 16:58:52.266789 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:52.266531 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-qtxbj" Sep 08 16:58:52.304654 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:52.304625 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-4j64x"] Sep 08 16:58:52.310764 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:52.310740 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-4j64x"] Sep 08 16:58:52.356213 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:52.356189 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-qtxbj"] Sep 08 16:58:52.360391 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:52.360369 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-qtxbj"] Sep 08 16:58:53.273219 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:53.273184 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d85785848-4jsdp" event={"ID":"1e2192da-d2a1-42c3-bbe3-31b1e061e306","Type":"ContainerStarted","Data":"5fb2e178bf854fafad5cb92a1c72403f44d5bfe927b264bf6c6ea0527e7cfd52"} Sep 08 16:58:53.293819 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:53.293766 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7d85785848-4jsdp" podStartSLOduration=1.347834967 podStartE2EDuration="2.29375122s" totalImagesPullingTime="945.916253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:58:53.290709098 +0000 UTC m=+496.027540215" watchObservedRunningTime="2026-09-08 16:58:53.29375122 +0000 UTC m=+496.030582375" Sep 08 16:58:53.332396 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:53.332363 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qmhp5"] Sep 08 16:58:53.332611 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:53.332590 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-qmhp5" podUID="e28aa243-4554-4ffc-b885-b1352a4b4439" containerName="authorino" containerID="cri-o://bfe75d885b6da89ce53a56d44618df78f45d134f6c48adb541954a764282c567" gracePeriod=30 Sep 08 16:58:53.575760 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:53.575738 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qmhp5" Sep 08 16:58:53.641838 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:53.641812 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wz6bx\" (UniqueName: \"kubernetes.io/projected/e28aa243-4554-4ffc-b885-b1352a4b4439-kube-api-access-wz6bx\") pod \"e28aa243-4554-4ffc-b885-b1352a4b4439\" (UID: \"e28aa243-4554-4ffc-b885-b1352a4b4439\") " Sep 08 16:58:53.643784 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:53.643762 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28aa243-4554-4ffc-b885-b1352a4b4439-kube-api-access-wz6bx" (OuterVolumeSpecName: "kube-api-access-wz6bx") pod "e28aa243-4554-4ffc-b885-b1352a4b4439" (UID: "e28aa243-4554-4ffc-b885-b1352a4b4439"). InnerVolumeSpecName "kube-api-access-wz6bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:58:53.742925 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:53.742886 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz6bx\" (UniqueName: \"kubernetes.io/projected/e28aa243-4554-4ffc-b885-b1352a4b4439-kube-api-access-wz6bx\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:58:53.759417 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:53.759392 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4acad7ef-98c3-4228-8e98-8729bff5c1e0" path="/var/lib/kubelet/pods/4acad7ef-98c3-4228-8e98-8729bff5c1e0/volumes" Sep 08 16:58:53.759664 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:53.759651 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="794b4bc4-ffa9-4432-bbc8-251d2cb55e48" path="/var/lib/kubelet/pods/794b4bc4-ffa9-4432-bbc8-251d2cb55e48/volumes" Sep 08 16:58:54.279152 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:54.279113 2565 generic.go:356] "Generic (PLEG): container finished" podID="e28aa243-4554-4ffc-b885-b1352a4b4439" containerID="bfe75d885b6da89ce53a56d44618df78f45d134f6c48adb541954a764282c567" exitCode=0 Sep 08 16:58:54.279152 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:54.279150 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qmhp5" Sep 08 16:58:54.279618 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:54.279196 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qmhp5" event={"ID":"e28aa243-4554-4ffc-b885-b1352a4b4439","Type":"ContainerDied","Data":"bfe75d885b6da89ce53a56d44618df78f45d134f6c48adb541954a764282c567"} Sep 08 16:58:54.279618 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:54.279232 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qmhp5" event={"ID":"e28aa243-4554-4ffc-b885-b1352a4b4439","Type":"ContainerDied","Data":"8db2c1f7745cc82573e19a78c9ebfe49b512b4d70bfd56d03cd5fea853b8aad7"} Sep 08 16:58:54.279618 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:54.279255 2565 scope.go:117] "RemoveContainer" containerID="bfe75d885b6da89ce53a56d44618df78f45d134f6c48adb541954a764282c567" Sep 08 16:58:54.288122 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:54.288106 2565 scope.go:117] "RemoveContainer" containerID="bfe75d885b6da89ce53a56d44618df78f45d134f6c48adb541954a764282c567" Sep 08 16:58:54.288366 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:58:54.288348 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfe75d885b6da89ce53a56d44618df78f45d134f6c48adb541954a764282c567\": container with ID starting with bfe75d885b6da89ce53a56d44618df78f45d134f6c48adb541954a764282c567 not found: ID does not exist" containerID="bfe75d885b6da89ce53a56d44618df78f45d134f6c48adb541954a764282c567" Sep 08 16:58:54.288417 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:54.288372 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe75d885b6da89ce53a56d44618df78f45d134f6c48adb541954a764282c567"} err="failed to get container status \"bfe75d885b6da89ce53a56d44618df78f45d134f6c48adb541954a764282c567\": rpc error: code = NotFound desc = could not find container \"bfe75d885b6da89ce53a56d44618df78f45d134f6c48adb541954a764282c567\": container with ID starting with bfe75d885b6da89ce53a56d44618df78f45d134f6c48adb541954a764282c567 not found: ID does not exist" Sep 08 16:58:54.298823 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:54.298798 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qmhp5"] Sep 08 16:58:54.302740 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:54.302721 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qmhp5"] Sep 08 16:58:55.760104 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:55.760070 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e28aa243-4554-4ffc-b885-b1352a4b4439" path="/var/lib/kubelet/pods/e28aa243-4554-4ffc-b885-b1352a4b4439/volumes" Sep 08 16:58:58.088691 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.088649 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:58:58.147383 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.147345 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-587bbc7cb-dcq2b"] Sep 08 16:58:58.147732 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.147719 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e28aa243-4554-4ffc-b885-b1352a4b4439" containerName="authorino" Sep 08 16:58:58.147796 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.147733 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e28aa243-4554-4ffc-b885-b1352a4b4439" containerName="authorino" Sep 08 16:58:58.147796 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.147794 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e28aa243-4554-4ffc-b885-b1352a4b4439" containerName="authorino" Sep 08 16:58:58.152033 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.152016 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587bbc7cb-dcq2b" Sep 08 16:58:58.154519 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.154491 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 08 16:58:58.154622 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.154601 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 08 16:58:58.154677 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.154623 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 08 16:58:58.155401 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.155386 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j7x8d\"" Sep 08 16:58:58.164268 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.164235 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-587bbc7cb-dcq2b"] Sep 08 16:58:58.204897 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.204854 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:58:58.279142 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.279117 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44fbbc41-35bb-4694-ad55-8c67dcec16d8-webhook-cert\") pod \"maas-controller-587bbc7cb-dcq2b\" (UID: \"44fbbc41-35bb-4694-ad55-8c67dcec16d8\") " pod="opendatahub/maas-controller-587bbc7cb-dcq2b" Sep 08 16:58:58.279264 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.279158 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44fbbc41-35bb-4694-ad55-8c67dcec16d8-metrics-certs\") pod \"maas-controller-587bbc7cb-dcq2b\" (UID: \"44fbbc41-35bb-4694-ad55-8c67dcec16d8\") " pod="opendatahub/maas-controller-587bbc7cb-dcq2b" Sep 08 16:58:58.279264 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.279179 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpbfx\" (UniqueName: \"kubernetes.io/projected/44fbbc41-35bb-4694-ad55-8c67dcec16d8-kube-api-access-kpbfx\") pod \"maas-controller-587bbc7cb-dcq2b\" (UID: \"44fbbc41-35bb-4694-ad55-8c67dcec16d8\") " pod="opendatahub/maas-controller-587bbc7cb-dcq2b" Sep 08 16:58:58.306718 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.306689 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:58:58.338084 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.338054 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75d5cb9587-vt7ng"] Sep 08 16:58:58.341727 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.341683 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75d5cb9587-vt7ng" Sep 08 16:58:58.352216 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.352192 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75d5cb9587-vt7ng"] Sep 08 16:58:58.379697 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.379664 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44fbbc41-35bb-4694-ad55-8c67dcec16d8-metrics-certs\") pod \"maas-controller-587bbc7cb-dcq2b\" (UID: \"44fbbc41-35bb-4694-ad55-8c67dcec16d8\") " pod="opendatahub/maas-controller-587bbc7cb-dcq2b" Sep 08 16:58:58.379697 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.379696 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpbfx\" (UniqueName: \"kubernetes.io/projected/44fbbc41-35bb-4694-ad55-8c67dcec16d8-kube-api-access-kpbfx\") pod \"maas-controller-587bbc7cb-dcq2b\" (UID: \"44fbbc41-35bb-4694-ad55-8c67dcec16d8\") " pod="opendatahub/maas-controller-587bbc7cb-dcq2b" Sep 08 16:58:58.379879 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.379786 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44fbbc41-35bb-4694-ad55-8c67dcec16d8-webhook-cert\") pod \"maas-controller-587bbc7cb-dcq2b\" (UID: \"44fbbc41-35bb-4694-ad55-8c67dcec16d8\") " pod="opendatahub/maas-controller-587bbc7cb-dcq2b" Sep 08 16:58:58.379944 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:58:58.379891 2565 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 08 16:58:58.380009 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:58:58.379946 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44fbbc41-35bb-4694-ad55-8c67dcec16d8-webhook-cert podName:44fbbc41-35bb-4694-ad55-8c67dcec16d8 nodeName:}" failed. No retries permitted until 2026-09-08 16:58:58.879927787 +0000 UTC m=+501.616758887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/44fbbc41-35bb-4694-ad55-8c67dcec16d8-webhook-cert") pod "maas-controller-587bbc7cb-dcq2b" (UID: "44fbbc41-35bb-4694-ad55-8c67dcec16d8") : secret "maas-controller-webhook-cert" not found Sep 08 16:58:58.382035 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.382009 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44fbbc41-35bb-4694-ad55-8c67dcec16d8-metrics-certs\") pod \"maas-controller-587bbc7cb-dcq2b\" (UID: \"44fbbc41-35bb-4694-ad55-8c67dcec16d8\") " pod="opendatahub/maas-controller-587bbc7cb-dcq2b" Sep 08 16:58:58.387908 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.387877 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpbfx\" (UniqueName: \"kubernetes.io/projected/44fbbc41-35bb-4694-ad55-8c67dcec16d8-kube-api-access-kpbfx\") pod \"maas-controller-587bbc7cb-dcq2b\" (UID: \"44fbbc41-35bb-4694-ad55-8c67dcec16d8\") " pod="opendatahub/maas-controller-587bbc7cb-dcq2b" Sep 08 16:58:58.480459 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.480416 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ccb45355-528f-453d-ad2d-210d4e8963ab-webhook-cert\") pod \"maas-controller-75d5cb9587-vt7ng\" (UID: \"ccb45355-528f-453d-ad2d-210d4e8963ab\") " pod="opendatahub/maas-controller-75d5cb9587-vt7ng" Sep 08 16:58:58.480542 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.480488 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccb45355-528f-453d-ad2d-210d4e8963ab-metrics-certs\") pod \"maas-controller-75d5cb9587-vt7ng\" (UID: \"ccb45355-528f-453d-ad2d-210d4e8963ab\") " pod="opendatahub/maas-controller-75d5cb9587-vt7ng" Sep 08 16:58:58.480542 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.480524 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n95x4\" (UniqueName: \"kubernetes.io/projected/ccb45355-528f-453d-ad2d-210d4e8963ab-kube-api-access-n95x4\") pod \"maas-controller-75d5cb9587-vt7ng\" (UID: \"ccb45355-528f-453d-ad2d-210d4e8963ab\") " pod="opendatahub/maas-controller-75d5cb9587-vt7ng" Sep 08 16:58:58.580936 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.580911 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ccb45355-528f-453d-ad2d-210d4e8963ab-webhook-cert\") pod \"maas-controller-75d5cb9587-vt7ng\" (UID: \"ccb45355-528f-453d-ad2d-210d4e8963ab\") " pod="opendatahub/maas-controller-75d5cb9587-vt7ng" Sep 08 16:58:58.581037 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.580950 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccb45355-528f-453d-ad2d-210d4e8963ab-metrics-certs\") pod \"maas-controller-75d5cb9587-vt7ng\" (UID: \"ccb45355-528f-453d-ad2d-210d4e8963ab\") " pod="opendatahub/maas-controller-75d5cb9587-vt7ng" Sep 08 16:58:58.581037 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.580981 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n95x4\" (UniqueName: \"kubernetes.io/projected/ccb45355-528f-453d-ad2d-210d4e8963ab-kube-api-access-n95x4\") pod \"maas-controller-75d5cb9587-vt7ng\" (UID: \"ccb45355-528f-453d-ad2d-210d4e8963ab\") " pod="opendatahub/maas-controller-75d5cb9587-vt7ng" Sep 08 16:58:58.583313 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.583285 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ccb45355-528f-453d-ad2d-210d4e8963ab-webhook-cert\") pod \"maas-controller-75d5cb9587-vt7ng\" (UID: \"ccb45355-528f-453d-ad2d-210d4e8963ab\") " pod="opendatahub/maas-controller-75d5cb9587-vt7ng" Sep 08 16:58:58.583398 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.583363 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccb45355-528f-453d-ad2d-210d4e8963ab-metrics-certs\") pod \"maas-controller-75d5cb9587-vt7ng\" (UID: \"ccb45355-528f-453d-ad2d-210d4e8963ab\") " pod="opendatahub/maas-controller-75d5cb9587-vt7ng" Sep 08 16:58:58.589257 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.589225 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n95x4\" (UniqueName: \"kubernetes.io/projected/ccb45355-528f-453d-ad2d-210d4e8963ab-kube-api-access-n95x4\") pod \"maas-controller-75d5cb9587-vt7ng\" (UID: \"ccb45355-528f-453d-ad2d-210d4e8963ab\") " pod="opendatahub/maas-controller-75d5cb9587-vt7ng" Sep 08 16:58:58.626594 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.626536 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:58:58.652092 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.652074 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75d5cb9587-vt7ng" Sep 08 16:58:58.883231 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.883149 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44fbbc41-35bb-4694-ad55-8c67dcec16d8-webhook-cert\") pod \"maas-controller-587bbc7cb-dcq2b\" (UID: \"44fbbc41-35bb-4694-ad55-8c67dcec16d8\") " pod="opendatahub/maas-controller-587bbc7cb-dcq2b" Sep 08 16:58:58.885445 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.885412 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44fbbc41-35bb-4694-ad55-8c67dcec16d8-webhook-cert\") pod \"maas-controller-587bbc7cb-dcq2b\" (UID: \"44fbbc41-35bb-4694-ad55-8c67dcec16d8\") " pod="opendatahub/maas-controller-587bbc7cb-dcq2b" Sep 08 16:58:58.924175 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:58.924147 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75d5cb9587-vt7ng"] Sep 08 16:58:58.925800 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:58:58.925778 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb45355_528f_453d_ad2d_210d4e8963ab.slice/crio-0233a1e08faca43a79eb7e5ebfa1d174da134ee9ea71beb72db5ccbf8aafcfc5 WatchSource:0}: Error finding container 0233a1e08faca43a79eb7e5ebfa1d174da134ee9ea71beb72db5ccbf8aafcfc5: Status 404 returned error can't find the container with id 0233a1e08faca43a79eb7e5ebfa1d174da134ee9ea71beb72db5ccbf8aafcfc5 Sep 08 16:58:59.062740 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:59.062719 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587bbc7cb-dcq2b" Sep 08 16:58:59.308691 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:59.308660 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75d5cb9587-vt7ng" event={"ID":"ccb45355-528f-453d-ad2d-210d4e8963ab","Type":"ContainerStarted","Data":"0233a1e08faca43a79eb7e5ebfa1d174da134ee9ea71beb72db5ccbf8aafcfc5"} Sep 08 16:58:59.337632 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:58:59.337610 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-587bbc7cb-dcq2b"] Sep 08 16:58:59.339158 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:58:59.339135 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44fbbc41_35bb_4694_ad55_8c67dcec16d8.slice/crio-2f76bf782c42a32b17a66dd7471dbf1e555c3a626f14bb8a9f42efd55ea278d0 WatchSource:0}: Error finding container 2f76bf782c42a32b17a66dd7471dbf1e555c3a626f14bb8a9f42efd55ea278d0: Status 404 returned error can't find the container with id 2f76bf782c42a32b17a66dd7471dbf1e555c3a626f14bb8a9f42efd55ea278d0 Sep 08 16:59:00.315503 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:00.315463 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587bbc7cb-dcq2b" event={"ID":"44fbbc41-35bb-4694-ad55-8c67dcec16d8","Type":"ContainerStarted","Data":"2f76bf782c42a32b17a66dd7471dbf1e555c3a626f14bb8a9f42efd55ea278d0"} Sep 08 16:59:01.087813 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:01.087724 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:59:01.192900 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:01.192870 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:59:01.323518 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:01.323481 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587bbc7cb-dcq2b" event={"ID":"44fbbc41-35bb-4694-ad55-8c67dcec16d8","Type":"ContainerStarted","Data":"0d9fd0c42ca907958e23bba8f5e8428d572b71c675316744f626da25ad54f356"} Sep 08 16:59:01.323975 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:01.323566 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-587bbc7cb-dcq2b" Sep 08 16:59:01.325227 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:01.325205 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75d5cb9587-vt7ng" event={"ID":"ccb45355-528f-453d-ad2d-210d4e8963ab","Type":"ContainerStarted","Data":"d4abcef5a7053b1ba59a4d2126741ab31f821a343007e72f575604ce5f9ab7ee"} Sep 08 16:59:01.325345 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:01.325303 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75d5cb9587-vt7ng" Sep 08 16:59:01.350635 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:01.350529 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-587bbc7cb-dcq2b" podStartSLOduration=1.8837983280000001 podStartE2EDuration="3.350510772s" totalImagesPullingTime="1.466712444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:59:01.348154475 +0000 UTC m=+504.084985597" watchObservedRunningTime="2026-09-08 16:59:01.350510772 +0000 UTC m=+504.087341884" Sep 08 16:59:01.382183 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:01.382129 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-75d5cb9587-vt7ng" podStartSLOduration=1.498591335 podStartE2EDuration="3.382111795s" totalImagesPullingTime="1.88352046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:59:01.380150915 +0000 UTC m=+504.116982032" watchObservedRunningTime="2026-09-08 16:59:01.382111795 +0000 UTC m=+504.118942910" Sep 08 16:59:12.333581 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:12.333545 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-587bbc7cb-dcq2b" Sep 08 16:59:12.333935 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:12.333709 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75d5cb9587-vt7ng" Sep 08 16:59:12.411534 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:12.411494 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-587bbc7cb-dcq2b"] Sep 08 16:59:12.411745 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:12.411719 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-587bbc7cb-dcq2b" podUID="44fbbc41-35bb-4694-ad55-8c67dcec16d8" containerName="manager" containerID="cri-o://0d9fd0c42ca907958e23bba8f5e8428d572b71c675316744f626da25ad54f356" gracePeriod=30 Sep 08 16:59:12.660478 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:12.660454 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587bbc7cb-dcq2b" Sep 08 16:59:12.798393 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:12.798363 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44fbbc41-35bb-4694-ad55-8c67dcec16d8-metrics-certs\") pod \"44fbbc41-35bb-4694-ad55-8c67dcec16d8\" (UID: \"44fbbc41-35bb-4694-ad55-8c67dcec16d8\") " Sep 08 16:59:12.798558 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:12.798403 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44fbbc41-35bb-4694-ad55-8c67dcec16d8-webhook-cert\") pod \"44fbbc41-35bb-4694-ad55-8c67dcec16d8\" (UID: \"44fbbc41-35bb-4694-ad55-8c67dcec16d8\") " Sep 08 16:59:12.798558 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:12.798445 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpbfx\" (UniqueName: \"kubernetes.io/projected/44fbbc41-35bb-4694-ad55-8c67dcec16d8-kube-api-access-kpbfx\") pod \"44fbbc41-35bb-4694-ad55-8c67dcec16d8\" (UID: \"44fbbc41-35bb-4694-ad55-8c67dcec16d8\") " Sep 08 16:59:12.800669 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:12.800640 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44fbbc41-35bb-4694-ad55-8c67dcec16d8-kube-api-access-kpbfx" (OuterVolumeSpecName: "kube-api-access-kpbfx") pod "44fbbc41-35bb-4694-ad55-8c67dcec16d8" (UID: "44fbbc41-35bb-4694-ad55-8c67dcec16d8"). InnerVolumeSpecName "kube-api-access-kpbfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:59:12.800770 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:12.800672 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44fbbc41-35bb-4694-ad55-8c67dcec16d8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "44fbbc41-35bb-4694-ad55-8c67dcec16d8" (UID: "44fbbc41-35bb-4694-ad55-8c67dcec16d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:59:12.800770 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:12.800694 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44fbbc41-35bb-4694-ad55-8c67dcec16d8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "44fbbc41-35bb-4694-ad55-8c67dcec16d8" (UID: "44fbbc41-35bb-4694-ad55-8c67dcec16d8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:59:12.899562 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:12.899496 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44fbbc41-35bb-4694-ad55-8c67dcec16d8-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:59:12.899562 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:12.899523 2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44fbbc41-35bb-4694-ad55-8c67dcec16d8-webhook-cert\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:59:12.899562 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:12.899532 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpbfx\" (UniqueName: \"kubernetes.io/projected/44fbbc41-35bb-4694-ad55-8c67dcec16d8-kube-api-access-kpbfx\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 16:59:13.392213 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:13.392177 2565 generic.go:356] "Generic (PLEG): container finished" podID="44fbbc41-35bb-4694-ad55-8c67dcec16d8" containerID="0d9fd0c42ca907958e23bba8f5e8428d572b71c675316744f626da25ad54f356" exitCode=0 Sep 08 16:59:13.392640 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:13.392240 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587bbc7cb-dcq2b" Sep 08 16:59:13.392640 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:13.392266 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587bbc7cb-dcq2b" event={"ID":"44fbbc41-35bb-4694-ad55-8c67dcec16d8","Type":"ContainerDied","Data":"0d9fd0c42ca907958e23bba8f5e8428d572b71c675316744f626da25ad54f356"} Sep 08 16:59:13.392640 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:13.392339 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587bbc7cb-dcq2b" event={"ID":"44fbbc41-35bb-4694-ad55-8c67dcec16d8","Type":"ContainerDied","Data":"2f76bf782c42a32b17a66dd7471dbf1e555c3a626f14bb8a9f42efd55ea278d0"} Sep 08 16:59:13.392640 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:13.392362 2565 scope.go:117] "RemoveContainer" containerID="0d9fd0c42ca907958e23bba8f5e8428d572b71c675316744f626da25ad54f356" Sep 08 16:59:13.401659 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:13.401643 2565 scope.go:117] "RemoveContainer" containerID="0d9fd0c42ca907958e23bba8f5e8428d572b71c675316744f626da25ad54f356" Sep 08 16:59:13.401915 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:59:13.401895 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9fd0c42ca907958e23bba8f5e8428d572b71c675316744f626da25ad54f356\": container with ID starting with 0d9fd0c42ca907958e23bba8f5e8428d572b71c675316744f626da25ad54f356 not found: ID does not exist" containerID="0d9fd0c42ca907958e23bba8f5e8428d572b71c675316744f626da25ad54f356" Sep 08 16:59:13.401965 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:13.401922 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9fd0c42ca907958e23bba8f5e8428d572b71c675316744f626da25ad54f356"} err="failed to get container status \"0d9fd0c42ca907958e23bba8f5e8428d572b71c675316744f626da25ad54f356\": rpc error: code = NotFound desc = could not find container \"0d9fd0c42ca907958e23bba8f5e8428d572b71c675316744f626da25ad54f356\": container with ID starting with 0d9fd0c42ca907958e23bba8f5e8428d572b71c675316744f626da25ad54f356 not found: ID does not exist" Sep 08 16:59:13.414284 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:13.414260 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-587bbc7cb-dcq2b"] Sep 08 16:59:13.421097 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:13.421075 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-587bbc7cb-dcq2b"] Sep 08 16:59:13.760661 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:13.760630 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44fbbc41-35bb-4694-ad55-8c67dcec16d8" path="/var/lib/kubelet/pods/44fbbc41-35bb-4694-ad55-8c67dcec16d8/volumes" Sep 08 16:59:21.411010 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:21.410978 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:59:33.678385 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:33.678292 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:59:33.771775 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:33.771739 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:59:35.246723 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.246682 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:59:35.352265 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.352234 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:59:35.403834 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.403797 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b"] Sep 08 16:59:35.404257 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.404238 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44fbbc41-35bb-4694-ad55-8c67dcec16d8" containerName="manager" Sep 08 16:59:35.404349 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.404259 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="44fbbc41-35bb-4694-ad55-8c67dcec16d8" containerName="manager" Sep 08 16:59:35.404349 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.404340 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="44fbbc41-35bb-4694-ad55-8c67dcec16d8" containerName="manager" Sep 08 16:59:35.408145 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.408124 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" Sep 08 16:59:35.410376 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.410346 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 08 16:59:35.410376 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.410368 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 08 16:59:35.411827 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.411810 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qpc2m\"" Sep 08 16:59:35.442914 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.442880 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b"] Sep 08 16:59:35.473395 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.473372 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-maas-api-tls\") pod \"maas-api-dd87ddf4c-lt57b\" (UID: \"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" Sep 08 16:59:35.473512 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.473400 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-openshift-service-ca\") pod \"maas-api-dd87ddf4c-lt57b\" (UID: \"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" Sep 08 16:59:35.473512 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.473491 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcsnk\" (UniqueName: \"kubernetes.io/projected/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-kube-api-access-jcsnk\") pod \"maas-api-dd87ddf4c-lt57b\" (UID: \"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" Sep 08 16:59:35.473588 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.473523 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-metrics-certs\") pod \"maas-api-dd87ddf4c-lt57b\" (UID: \"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" Sep 08 16:59:35.574040 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.573974 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcsnk\" (UniqueName: \"kubernetes.io/projected/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-kube-api-access-jcsnk\") pod \"maas-api-dd87ddf4c-lt57b\" (UID: \"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" Sep 08 16:59:35.574040 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.574012 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-metrics-certs\") pod \"maas-api-dd87ddf4c-lt57b\" (UID: \"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" Sep 08 16:59:35.574186 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.574142 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-maas-api-tls\") pod \"maas-api-dd87ddf4c-lt57b\" (UID: \"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" Sep 08 16:59:35.574186 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.574178 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-openshift-service-ca\") pod \"maas-api-dd87ddf4c-lt57b\" (UID: \"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" Sep 08 16:59:35.574287 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:59:35.574269 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Sep 08 16:59:35.574356 ip-10-0-137-108 kubenswrapper[2565]: E0908 16:59:35.574336 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-maas-api-tls podName:ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2 nodeName:}" failed. No retries permitted until 2026-09-08 16:59:36.074319931 +0000 UTC m=+538.811151024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-maas-api-tls") pod "maas-api-dd87ddf4c-lt57b" (UID: "ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2") : secret "maas-api-serving-cert" not found Sep 08 16:59:35.574799 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.574777 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-openshift-service-ca\") pod \"maas-api-dd87ddf4c-lt57b\" (UID: \"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" Sep 08 16:59:35.576533 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.576511 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-metrics-certs\") pod \"maas-api-dd87ddf4c-lt57b\" (UID: \"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" Sep 08 16:59:35.585250 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.585227 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcsnk\" (UniqueName: \"kubernetes.io/projected/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-kube-api-access-jcsnk\") pod \"maas-api-dd87ddf4c-lt57b\" (UID: \"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" Sep 08 16:59:35.679188 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.679157 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:59:35.828922 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:35.828838 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:59:36.078660 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:36.078622 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-maas-api-tls\") pod \"maas-api-dd87ddf4c-lt57b\" (UID: \"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" Sep 08 16:59:36.081068 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:36.081013 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-maas-api-tls\") pod \"maas-api-dd87ddf4c-lt57b\" (UID: \"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" Sep 08 16:59:36.105411 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:36.105375 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:59:36.201928 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:36.201895 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:59:36.318112 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:36.318078 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" Sep 08 16:59:36.394656 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:36.394627 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-vrwnc"] Sep 08 16:59:36.400864 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:36.400841 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-vrwnc" Sep 08 16:59:36.403471 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:36.403440 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 08 16:59:36.403921 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:36.403899 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-z9jf4\"" Sep 08 16:59:36.409493 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:36.409009 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-vrwnc"] Sep 08 16:59:36.482708 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:36.482666 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5cd24d0-2d8f-4639-9fac-d4ae2ba1676d-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-vrwnc\" (UID: \"f5cd24d0-2d8f-4639-9fac-d4ae2ba1676d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-vrwnc" Sep 08 16:59:36.482901 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:36.482730 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np62k\" (UniqueName: \"kubernetes.io/projected/f5cd24d0-2d8f-4639-9fac-d4ae2ba1676d-kube-api-access-np62k\") pod \"payload-processing-5b54b7fbb7-vrwnc\" (UID: \"f5cd24d0-2d8f-4639-9fac-d4ae2ba1676d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-vrwnc" Sep 08 16:59:36.584037 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:36.583949 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np62k\" (UniqueName: \"kubernetes.io/projected/f5cd24d0-2d8f-4639-9fac-d4ae2ba1676d-kube-api-access-np62k\") pod \"payload-processing-5b54b7fbb7-vrwnc\" (UID: \"f5cd24d0-2d8f-4639-9fac-d4ae2ba1676d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-vrwnc" Sep 08 16:59:36.584206 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:36.584114 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5cd24d0-2d8f-4639-9fac-d4ae2ba1676d-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-vrwnc\" (UID: \"f5cd24d0-2d8f-4639-9fac-d4ae2ba1676d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-vrwnc" Sep 08 16:59:36.585010 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:36.584979 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5cd24d0-2d8f-4639-9fac-d4ae2ba1676d-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-vrwnc\" (UID: \"f5cd24d0-2d8f-4639-9fac-d4ae2ba1676d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-vrwnc" Sep 08 16:59:36.592599 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:36.592563 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np62k\" (UniqueName: \"kubernetes.io/projected/f5cd24d0-2d8f-4639-9fac-d4ae2ba1676d-kube-api-access-np62k\") pod \"payload-processing-5b54b7fbb7-vrwnc\" (UID: \"f5cd24d0-2d8f-4639-9fac-d4ae2ba1676d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-vrwnc" Sep 08 16:59:36.601565 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:36.601539 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b"] Sep 08 16:59:36.602734 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:59:36.602711 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea64a9f3_4c5d_4ba4_9535_cd0f4774e9e2.slice/crio-cb3ff11a2d606a652441c8f22dd5475ff75b8d1102bf74a699f4853f886dad4c WatchSource:0}: Error finding container cb3ff11a2d606a652441c8f22dd5475ff75b8d1102bf74a699f4853f886dad4c: Status 404 returned error can't find the container with id cb3ff11a2d606a652441c8f22dd5475ff75b8d1102bf74a699f4853f886dad4c Sep 08 16:59:36.716626 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:36.716596 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-vrwnc" Sep 08 16:59:36.995857 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:36.995831 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-vrwnc"] Sep 08 16:59:36.997259 ip-10-0-137-108 kubenswrapper[2565]: W0908 16:59:36.997232 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cd24d0_2d8f_4639_9fac_d4ae2ba1676d.slice/crio-036aed5d8d30a9c38001231b221f4d4904252fc675cd5cf8827dcd631af719ef WatchSource:0}: Error finding container 036aed5d8d30a9c38001231b221f4d4904252fc675cd5cf8827dcd631af719ef: Status 404 returned error can't find the container with id 036aed5d8d30a9c38001231b221f4d4904252fc675cd5cf8827dcd631af719ef Sep 08 16:59:37.537974 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:37.537874 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" event={"ID":"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2","Type":"ContainerStarted","Data":"cb3ff11a2d606a652441c8f22dd5475ff75b8d1102bf74a699f4853f886dad4c"} Sep 08 16:59:37.540413 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:37.540384 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-vrwnc" event={"ID":"f5cd24d0-2d8f-4639-9fac-d4ae2ba1676d","Type":"ContainerStarted","Data":"036aed5d8d30a9c38001231b221f4d4904252fc675cd5cf8827dcd631af719ef"} Sep 08 16:59:39.560727 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:39.560692 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" event={"ID":"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2","Type":"ContainerStarted","Data":"dc19028dfb187f19f88167356eb2a556a01c07c07cbc2315785b6470ea3c552b"} Sep 08 16:59:39.561158 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:39.560812 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" Sep 08 16:59:39.562355 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:39.562329 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-vrwnc" event={"ID":"f5cd24d0-2d8f-4639-9fac-d4ae2ba1676d","Type":"ContainerStarted","Data":"b91e9553ceaf36069ef42d7bb50f17f5d79b9463addfefbed9853b06ad53e746"} Sep 08 16:59:39.562490 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:39.562470 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-vrwnc" Sep 08 16:59:39.590680 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:39.590634 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" podStartSLOduration=2.250536684 podStartE2EDuration="4.590620822s" totalImagesPullingTime="2.340084138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:59:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:59:39.588409551 +0000 UTC m=+542.325240667" watchObservedRunningTime="2026-09-08 16:59:39.590620822 +0000 UTC m=+542.327451938" Sep 08 16:59:45.573240 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:45.573208 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" Sep 08 16:59:45.594905 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:45.594857 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-vrwnc" podStartSLOduration=7.641032344 podStartE2EDuration="9.59484177s" totalImagesPullingTime="1.953809426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:59:39.615404919 +0000 UTC m=+542.352236034" watchObservedRunningTime="2026-09-08 16:59:45.59484177 +0000 UTC m=+548.331672886" Sep 08 16:59:46.598010 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:46.597973 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:59:46.699357 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:46.699325 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 16:59:50.570976 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:50.570946 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-vrwnc" Sep 08 16:59:51.270107 ip-10-0-137-108 kubenswrapper[2565]: I0908 16:59:51.270069 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:00:03.257254 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:03.257211 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:00:14.315364 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.315332 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lt675"] Sep 08 17:00:14.322072 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.322053 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:14.324510 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.324490 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 08 17:00:14.325221 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.325202 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 08 17:00:14.325273 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.325236 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 08 17:00:14.325318 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.325269 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9fdqs\"" Sep 08 17:00:14.329048 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.328904 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lt675"] Sep 08 17:00:14.348245 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.348219 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:00:14.500187 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.500153 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e19ac764-d081-445b-a44c-c77ca448cdbe-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lt675\" (UID: \"e19ac764-d081-445b-a44c-c77ca448cdbe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:14.500187 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.500190 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwp6f\" (UniqueName: \"kubernetes.io/projected/e19ac764-d081-445b-a44c-c77ca448cdbe-kube-api-access-qwp6f\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lt675\" (UID: \"e19ac764-d081-445b-a44c-c77ca448cdbe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:14.500392 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.500211 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e19ac764-d081-445b-a44c-c77ca448cdbe-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lt675\" (UID: \"e19ac764-d081-445b-a44c-c77ca448cdbe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:14.500392 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.500302 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e19ac764-d081-445b-a44c-c77ca448cdbe-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lt675\" (UID: \"e19ac764-d081-445b-a44c-c77ca448cdbe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:14.500392 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.500380 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e19ac764-d081-445b-a44c-c77ca448cdbe-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lt675\" (UID: \"e19ac764-d081-445b-a44c-c77ca448cdbe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:14.500528 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.500407 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e19ac764-d081-445b-a44c-c77ca448cdbe-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lt675\" (UID: \"e19ac764-d081-445b-a44c-c77ca448cdbe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:14.601583 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.601499 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e19ac764-d081-445b-a44c-c77ca448cdbe-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lt675\" (UID: \"e19ac764-d081-445b-a44c-c77ca448cdbe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:14.601583 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.601565 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e19ac764-d081-445b-a44c-c77ca448cdbe-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lt675\" (UID: \"e19ac764-d081-445b-a44c-c77ca448cdbe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:14.601804 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.601588 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e19ac764-d081-445b-a44c-c77ca448cdbe-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lt675\" (UID: \"e19ac764-d081-445b-a44c-c77ca448cdbe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:14.601804 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.601624 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e19ac764-d081-445b-a44c-c77ca448cdbe-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lt675\" (UID: \"e19ac764-d081-445b-a44c-c77ca448cdbe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:14.601804 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.601645 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwp6f\" (UniqueName: \"kubernetes.io/projected/e19ac764-d081-445b-a44c-c77ca448cdbe-kube-api-access-qwp6f\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lt675\" (UID: \"e19ac764-d081-445b-a44c-c77ca448cdbe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:14.601804 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.601666 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e19ac764-d081-445b-a44c-c77ca448cdbe-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lt675\" (UID: \"e19ac764-d081-445b-a44c-c77ca448cdbe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:14.602036 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.601938 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e19ac764-d081-445b-a44c-c77ca448cdbe-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lt675\" (UID: \"e19ac764-d081-445b-a44c-c77ca448cdbe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:14.602036 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.602019 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e19ac764-d081-445b-a44c-c77ca448cdbe-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lt675\" (UID: \"e19ac764-d081-445b-a44c-c77ca448cdbe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:14.602115 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.602056 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e19ac764-d081-445b-a44c-c77ca448cdbe-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lt675\" (UID: \"e19ac764-d081-445b-a44c-c77ca448cdbe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:14.603881 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.603862 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e19ac764-d081-445b-a44c-c77ca448cdbe-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lt675\" (UID: \"e19ac764-d081-445b-a44c-c77ca448cdbe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:14.604210 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.604193 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e19ac764-d081-445b-a44c-c77ca448cdbe-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lt675\" (UID: \"e19ac764-d081-445b-a44c-c77ca448cdbe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:14.610251 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.610226 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwp6f\" (UniqueName: \"kubernetes.io/projected/e19ac764-d081-445b-a44c-c77ca448cdbe-kube-api-access-qwp6f\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lt675\" (UID: \"e19ac764-d081-445b-a44c-c77ca448cdbe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:14.632841 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.632820 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:14.910778 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:14.910750 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lt675"] Sep 08 17:00:14.913480 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:00:14.913453 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19ac764_d081_445b_a44c_c77ca448cdbe.slice/crio-934f4881554d82b971013ab28c5df07d602d0e4867d5badd6b85fb6825f72616 WatchSource:0}: Error finding container 934f4881554d82b971013ab28c5df07d602d0e4867d5badd6b85fb6825f72616: Status 404 returned error can't find the container with id 934f4881554d82b971013ab28c5df07d602d0e4867d5badd6b85fb6825f72616 Sep 08 17:00:15.766329 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:15.766296 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" event={"ID":"e19ac764-d081-445b-a44c-c77ca448cdbe","Type":"ContainerStarted","Data":"934f4881554d82b971013ab28c5df07d602d0e4867d5badd6b85fb6825f72616"} Sep 08 17:00:21.800753 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:21.800718 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" event={"ID":"e19ac764-d081-445b-a44c-c77ca448cdbe","Type":"ContainerStarted","Data":"e91667c2afed841f4f210e1f0db5651e2d5572c133f5a68f114d7495e1df0164"} Sep 08 17:00:21.822996 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:21.822952 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" podStartSLOduration=1.4322151220000001 podStartE2EDuration="7.822936585s" totalImagesPullingTime="6.390721463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:00:21.820419604 +0000 UTC m=+584.557271845" watchObservedRunningTime="2026-09-08 17:00:21.822936585 +0000 UTC m=+584.559767701" Sep 08 17:00:24.633877 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:24.633829 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:24.634274 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:24.633900 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:24.646396 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:24.646371 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:24.828297 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:24.828269 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lt675" Sep 08 17:00:27.558899 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:27.558864 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:00:37.734507 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:37.734479 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/2.log" Sep 08 17:00:37.736718 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:37.736693 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/2.log" Sep 08 17:00:37.742442 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:37.742390 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xxnl_bf53ac63-f864-402b-bf12-45b289e7e13d/ovn-acl-logging/0.log" Sep 08 17:00:37.744760 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:37.744742 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xxnl_bf53ac63-f864-402b-bf12-45b289e7e13d/ovn-acl-logging/0.log" Sep 08 17:00:38.899129 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:00:38.899089 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:01:01.135026 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:01.134941 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:01:05.607223 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:05.607183 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:01:10.220947 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.220912 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf"] Sep 08 17:01:10.224504 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.224483 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:10.228182 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.228156 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 08 17:01:10.236891 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.236866 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf"] Sep 08 17:01:10.271264 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.271192 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:01:10.357221 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.357187 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b011a03-62bd-473e-bbb6-4863b92ad9d8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-22pzf\" (UID: \"0b011a03-62bd-473e-bbb6-4863b92ad9d8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:10.357387 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.357238 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b011a03-62bd-473e-bbb6-4863b92ad9d8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-22pzf\" (UID: \"0b011a03-62bd-473e-bbb6-4863b92ad9d8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:10.357387 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.357277 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b011a03-62bd-473e-bbb6-4863b92ad9d8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-22pzf\" (UID: \"0b011a03-62bd-473e-bbb6-4863b92ad9d8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:10.357387 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.357330 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b011a03-62bd-473e-bbb6-4863b92ad9d8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-22pzf\" (UID: \"0b011a03-62bd-473e-bbb6-4863b92ad9d8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:10.357548 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.357392 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxp6n\" (UniqueName: \"kubernetes.io/projected/0b011a03-62bd-473e-bbb6-4863b92ad9d8-kube-api-access-lxp6n\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-22pzf\" (UID: \"0b011a03-62bd-473e-bbb6-4863b92ad9d8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:10.357548 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.357444 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b011a03-62bd-473e-bbb6-4863b92ad9d8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-22pzf\" (UID: \"0b011a03-62bd-473e-bbb6-4863b92ad9d8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:10.458079 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.458039 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b011a03-62bd-473e-bbb6-4863b92ad9d8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-22pzf\" (UID: \"0b011a03-62bd-473e-bbb6-4863b92ad9d8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:10.458252 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.458088 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b011a03-62bd-473e-bbb6-4863b92ad9d8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-22pzf\" (UID: \"0b011a03-62bd-473e-bbb6-4863b92ad9d8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:10.458252 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.458150 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b011a03-62bd-473e-bbb6-4863b92ad9d8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-22pzf\" (UID: \"0b011a03-62bd-473e-bbb6-4863b92ad9d8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:10.458252 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.458184 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxp6n\" (UniqueName: \"kubernetes.io/projected/0b011a03-62bd-473e-bbb6-4863b92ad9d8-kube-api-access-lxp6n\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-22pzf\" (UID: \"0b011a03-62bd-473e-bbb6-4863b92ad9d8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:10.458407 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.458315 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b011a03-62bd-473e-bbb6-4863b92ad9d8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-22pzf\" (UID: \"0b011a03-62bd-473e-bbb6-4863b92ad9d8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:10.458831 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.458405 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b011a03-62bd-473e-bbb6-4863b92ad9d8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-22pzf\" (UID: \"0b011a03-62bd-473e-bbb6-4863b92ad9d8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:10.458831 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.458559 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b011a03-62bd-473e-bbb6-4863b92ad9d8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-22pzf\" (UID: \"0b011a03-62bd-473e-bbb6-4863b92ad9d8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:10.458831 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.458611 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b011a03-62bd-473e-bbb6-4863b92ad9d8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-22pzf\" (UID: \"0b011a03-62bd-473e-bbb6-4863b92ad9d8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:10.458831 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.458757 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b011a03-62bd-473e-bbb6-4863b92ad9d8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-22pzf\" (UID: \"0b011a03-62bd-473e-bbb6-4863b92ad9d8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:10.460573 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.460552 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b011a03-62bd-473e-bbb6-4863b92ad9d8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-22pzf\" (UID: \"0b011a03-62bd-473e-bbb6-4863b92ad9d8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:10.461023 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.461007 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b011a03-62bd-473e-bbb6-4863b92ad9d8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-22pzf\" (UID: \"0b011a03-62bd-473e-bbb6-4863b92ad9d8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:10.467631 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.467605 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxp6n\" (UniqueName: \"kubernetes.io/projected/0b011a03-62bd-473e-bbb6-4863b92ad9d8-kube-api-access-lxp6n\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-22pzf\" (UID: \"0b011a03-62bd-473e-bbb6-4863b92ad9d8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:10.536756 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.536685 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:10.819035 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.819009 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf"] Sep 08 17:01:10.820372 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:01:10.820341 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b011a03_62bd_473e_bbb6_4863b92ad9d8.slice/crio-3a90ef44ce5af96e7c628d8542bb30f4365e2095ba706074f777e49697f9914b WatchSource:0}: Error finding container 3a90ef44ce5af96e7c628d8542bb30f4365e2095ba706074f777e49697f9914b: Status 404 returned error can't find the container with id 3a90ef44ce5af96e7c628d8542bb30f4365e2095ba706074f777e49697f9914b Sep 08 17:01:10.821824 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:10.821804 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 17:01:11.078350 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:11.078322 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" event={"ID":"0b011a03-62bd-473e-bbb6-4863b92ad9d8","Type":"ContainerStarted","Data":"3a90ef44ce5af96e7c628d8542bb30f4365e2095ba706074f777e49697f9914b"} Sep 08 17:01:12.086339 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:12.086294 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" event={"ID":"0b011a03-62bd-473e-bbb6-4863b92ad9d8","Type":"ContainerStarted","Data":"14b3ace01ab07782c29c4bee5df6b02e486022a16eb19c3db19f86c335f680c8"} Sep 08 17:01:12.110071 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:12.110011 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" podStartSLOduration=1.907968251 podStartE2EDuration="2.109990905s" totalImagesPullingTime="202.022654ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:01:12.108373416 +0000 UTC m=+634.845204533" watchObservedRunningTime="2026-09-08 17:01:12.109990905 +0000 UTC m=+634.846822022" Sep 08 17:01:20.537748 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:20.537713 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:20.538124 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:20.537774 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:20.550545 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:20.550520 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:21.150266 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:21.150230 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-22pzf" Sep 08 17:01:30.197754 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:30.197704 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:01:34.738906 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:34.738863 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:01:49.799749 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:49.799716 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:01:49.887481 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:01:49.887448 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:02:01.537395 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:01.537351 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:02:22.805281 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:22.805247 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:02:37.833671 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:37.833588 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-755dd4cb9c-7nrw6"] Sep 08 17:02:37.837281 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:37.837260 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-755dd4cb9c-7nrw6" Sep 08 17:02:37.854135 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:37.854109 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-755dd4cb9c-7nrw6"] Sep 08 17:02:37.973149 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:37.973110 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc7ll\" (UniqueName: \"kubernetes.io/projected/5693acfa-95c2-4930-bf0f-c56c38438c7d-kube-api-access-tc7ll\") pod \"authorino-755dd4cb9c-7nrw6\" (UID: \"5693acfa-95c2-4930-bf0f-c56c38438c7d\") " pod="kuadrant-system/authorino-755dd4cb9c-7nrw6" Sep 08 17:02:37.973310 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:37.973163 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5693acfa-95c2-4930-bf0f-c56c38438c7d-tls-cert\") pod \"authorino-755dd4cb9c-7nrw6\" (UID: \"5693acfa-95c2-4930-bf0f-c56c38438c7d\") " pod="kuadrant-system/authorino-755dd4cb9c-7nrw6" Sep 08 17:02:38.074095 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:38.074057 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc7ll\" (UniqueName: \"kubernetes.io/projected/5693acfa-95c2-4930-bf0f-c56c38438c7d-kube-api-access-tc7ll\") pod \"authorino-755dd4cb9c-7nrw6\" (UID: \"5693acfa-95c2-4930-bf0f-c56c38438c7d\") " pod="kuadrant-system/authorino-755dd4cb9c-7nrw6" Sep 08 17:02:38.074298 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:38.074139 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5693acfa-95c2-4930-bf0f-c56c38438c7d-tls-cert\") pod \"authorino-755dd4cb9c-7nrw6\" (UID: \"5693acfa-95c2-4930-bf0f-c56c38438c7d\") " pod="kuadrant-system/authorino-755dd4cb9c-7nrw6" Sep 08 17:02:38.077035 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:38.077005 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5693acfa-95c2-4930-bf0f-c56c38438c7d-tls-cert\") pod \"authorino-755dd4cb9c-7nrw6\" (UID: \"5693acfa-95c2-4930-bf0f-c56c38438c7d\") " pod="kuadrant-system/authorino-755dd4cb9c-7nrw6" Sep 08 17:02:38.083709 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:38.083647 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc7ll\" (UniqueName: \"kubernetes.io/projected/5693acfa-95c2-4930-bf0f-c56c38438c7d-kube-api-access-tc7ll\") pod \"authorino-755dd4cb9c-7nrw6\" (UID: \"5693acfa-95c2-4930-bf0f-c56c38438c7d\") " pod="kuadrant-system/authorino-755dd4cb9c-7nrw6" Sep 08 17:02:38.146740 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:38.146696 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-755dd4cb9c-7nrw6" Sep 08 17:02:38.422470 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:38.422443 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-755dd4cb9c-7nrw6"] Sep 08 17:02:38.424152 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:02:38.424125 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5693acfa_95c2_4930_bf0f_c56c38438c7d.slice/crio-567678c977fd7fb6d9fe87fe6fd03af9a9283465b0b0dabd9f65d6391ebf901e WatchSource:0}: Error finding container 567678c977fd7fb6d9fe87fe6fd03af9a9283465b0b0dabd9f65d6391ebf901e: Status 404 returned error can't find the container with id 567678c977fd7fb6d9fe87fe6fd03af9a9283465b0b0dabd9f65d6391ebf901e Sep 08 17:02:38.585529 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:38.585482 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-755dd4cb9c-7nrw6" event={"ID":"5693acfa-95c2-4930-bf0f-c56c38438c7d","Type":"ContainerStarted","Data":"567678c977fd7fb6d9fe87fe6fd03af9a9283465b0b0dabd9f65d6391ebf901e"} Sep 08 17:02:39.593617 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:39.593579 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-755dd4cb9c-7nrw6" event={"ID":"5693acfa-95c2-4930-bf0f-c56c38438c7d","Type":"ContainerStarted","Data":"a26dded912db9be8f01f18c0c4a14b56d2249b08eea23bea9aad85c3f8d501b2"} Sep 08 17:02:39.618823 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:39.618764 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-755dd4cb9c-7nrw6" podStartSLOduration=1.574314163 podStartE2EDuration="2.618747548s" totalImagesPullingTime="1.044433385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:02:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:02:39.61713782 +0000 UTC m=+722.353968937" watchObservedRunningTime="2026-09-08 17:02:39.618747548 +0000 UTC m=+722.355578664" Sep 08 17:02:39.667349 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:39.667311 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d85785848-4jsdp"] Sep 08 17:02:39.667618 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:39.667569 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7d85785848-4jsdp" podUID="1e2192da-d2a1-42c3-bbe3-31b1e061e306" containerName="authorino" containerID="cri-o://5fb2e178bf854fafad5cb92a1c72403f44d5bfe927b264bf6c6ea0527e7cfd52" gracePeriod=30 Sep 08 17:02:39.908698 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:39.908659 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d85785848-4jsdp" Sep 08 17:02:39.990606 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:39.990565 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1e2192da-d2a1-42c3-bbe3-31b1e061e306-tls-cert\") pod \"1e2192da-d2a1-42c3-bbe3-31b1e061e306\" (UID: \"1e2192da-d2a1-42c3-bbe3-31b1e061e306\") " Sep 08 17:02:39.990808 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:39.990657 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22q42\" (UniqueName: \"kubernetes.io/projected/1e2192da-d2a1-42c3-bbe3-31b1e061e306-kube-api-access-22q42\") pod \"1e2192da-d2a1-42c3-bbe3-31b1e061e306\" (UID: \"1e2192da-d2a1-42c3-bbe3-31b1e061e306\") " Sep 08 17:02:39.992784 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:39.992744 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2192da-d2a1-42c3-bbe3-31b1e061e306-kube-api-access-22q42" (OuterVolumeSpecName: "kube-api-access-22q42") pod "1e2192da-d2a1-42c3-bbe3-31b1e061e306" (UID: "1e2192da-d2a1-42c3-bbe3-31b1e061e306"). InnerVolumeSpecName "kube-api-access-22q42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:02:40.002489 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:40.002450 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e2192da-d2a1-42c3-bbe3-31b1e061e306-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "1e2192da-d2a1-42c3-bbe3-31b1e061e306" (UID: "1e2192da-d2a1-42c3-bbe3-31b1e061e306"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:02:40.091695 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:40.091598 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22q42\" (UniqueName: \"kubernetes.io/projected/1e2192da-d2a1-42c3-bbe3-31b1e061e306-kube-api-access-22q42\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:02:40.091695 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:40.091639 2565 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1e2192da-d2a1-42c3-bbe3-31b1e061e306-tls-cert\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:02:40.233712 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:40.233676 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:02:40.327608 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:40.327570 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:02:40.601764 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:40.601726 2565 generic.go:356] "Generic (PLEG): container finished" podID="1e2192da-d2a1-42c3-bbe3-31b1e061e306" containerID="5fb2e178bf854fafad5cb92a1c72403f44d5bfe927b264bf6c6ea0527e7cfd52" exitCode=0 Sep 08 17:02:40.602191 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:40.601797 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d85785848-4jsdp" Sep 08 17:02:40.602191 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:40.601811 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d85785848-4jsdp" event={"ID":"1e2192da-d2a1-42c3-bbe3-31b1e061e306","Type":"ContainerDied","Data":"5fb2e178bf854fafad5cb92a1c72403f44d5bfe927b264bf6c6ea0527e7cfd52"} Sep 08 17:02:40.602191 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:40.601859 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d85785848-4jsdp" event={"ID":"1e2192da-d2a1-42c3-bbe3-31b1e061e306","Type":"ContainerDied","Data":"2f1858e06ccd375529eb1e56c8ddbd5aae2ce9975a070c0f7d2a846a9021415f"} Sep 08 17:02:40.602191 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:40.601880 2565 scope.go:117] "RemoveContainer" containerID="5fb2e178bf854fafad5cb92a1c72403f44d5bfe927b264bf6c6ea0527e7cfd52" Sep 08 17:02:40.611858 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:40.611839 2565 scope.go:117] "RemoveContainer" containerID="5fb2e178bf854fafad5cb92a1c72403f44d5bfe927b264bf6c6ea0527e7cfd52" Sep 08 17:02:40.612186 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:02:40.612165 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb2e178bf854fafad5cb92a1c72403f44d5bfe927b264bf6c6ea0527e7cfd52\": container with ID starting with 5fb2e178bf854fafad5cb92a1c72403f44d5bfe927b264bf6c6ea0527e7cfd52 not found: ID does not exist" containerID="5fb2e178bf854fafad5cb92a1c72403f44d5bfe927b264bf6c6ea0527e7cfd52" Sep 08 17:02:40.612268 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:40.612198 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb2e178bf854fafad5cb92a1c72403f44d5bfe927b264bf6c6ea0527e7cfd52"} err="failed to get container status \"5fb2e178bf854fafad5cb92a1c72403f44d5bfe927b264bf6c6ea0527e7cfd52\": rpc error: code = NotFound desc = could not find container \"5fb2e178bf854fafad5cb92a1c72403f44d5bfe927b264bf6c6ea0527e7cfd52\": container with ID starting with 5fb2e178bf854fafad5cb92a1c72403f44d5bfe927b264bf6c6ea0527e7cfd52 not found: ID does not exist" Sep 08 17:02:40.632556 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:40.632525 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d85785848-4jsdp"] Sep 08 17:02:40.647128 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:40.647085 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7d85785848-4jsdp"] Sep 08 17:02:41.453704 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:41.453668 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:02:41.549816 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:41.549766 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:02:41.759957 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:41.759882 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e2192da-d2a1-42c3-bbe3-31b1e061e306" path="/var/lib/kubelet/pods/1e2192da-d2a1-42c3-bbe3-31b1e061e306/volumes" Sep 08 17:02:58.364175 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:58.364124 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:02:58.613950 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:58.613893 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:02:59.122817 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:59.122781 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:02:59.356712 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:02:59.356678 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:00.239494 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:00.239454 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:00.510626 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:00.510549 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:00.800281 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:00.800206 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7841-9d2b7c"] Sep 08 17:03:00.800623 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:00.800609 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e2192da-d2a1-42c3-bbe3-31b1e061e306" containerName="authorino" Sep 08 17:03:00.800623 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:00.800624 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e2192da-d2a1-42c3-bbe3-31b1e061e306" containerName="authorino" Sep 08 17:03:00.800726 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:00.800694 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e2192da-d2a1-42c3-bbe3-31b1e061e306" containerName="authorino" Sep 08 17:03:00.804883 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:00.804866 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7841-9d2b7c" Sep 08 17:03:00.809480 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:00.809459 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v8zrb\"" Sep 08 17:03:00.821505 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:00.821480 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7841-9d2b7c"] Sep 08 17:03:00.857385 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:00.857357 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtndg\" (UniqueName: \"kubernetes.io/projected/822c04a4-eae9-48d2-b255-852696fdfdc6-kube-api-access-mtndg\") pod \"test-curl-7841-9d2b7c\" (UID: \"822c04a4-eae9-48d2-b255-852696fdfdc6\") " pod="opendatahub/test-curl-7841-9d2b7c" Sep 08 17:03:00.958539 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:00.958509 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtndg\" (UniqueName: \"kubernetes.io/projected/822c04a4-eae9-48d2-b255-852696fdfdc6-kube-api-access-mtndg\") pod \"test-curl-7841-9d2b7c\" (UID: \"822c04a4-eae9-48d2-b255-852696fdfdc6\") " pod="opendatahub/test-curl-7841-9d2b7c" Sep 08 17:03:00.979828 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:00.979800 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtndg\" (UniqueName: \"kubernetes.io/projected/822c04a4-eae9-48d2-b255-852696fdfdc6-kube-api-access-mtndg\") pod \"test-curl-7841-9d2b7c\" (UID: \"822c04a4-eae9-48d2-b255-852696fdfdc6\") " pod="opendatahub/test-curl-7841-9d2b7c" Sep 08 17:03:01.114181 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:01.114112 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7841-9d2b7c" Sep 08 17:03:01.407899 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:01.407871 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7841-9d2b7c"] Sep 08 17:03:01.409808 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:03:01.409775 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod822c04a4_eae9_48d2_b255_852696fdfdc6.slice/crio-76d8ef854cb9c18ffca46f278f80d369e9ff0b688f97f96a1325c0a0189737b7 WatchSource:0}: Error finding container 76d8ef854cb9c18ffca46f278f80d369e9ff0b688f97f96a1325c0a0189737b7: Status 404 returned error can't find the container with id 76d8ef854cb9c18ffca46f278f80d369e9ff0b688f97f96a1325c0a0189737b7 Sep 08 17:03:01.725534 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:01.725488 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7841-9d2b7c" event={"ID":"822c04a4-eae9-48d2-b255-852696fdfdc6","Type":"ContainerStarted","Data":"76d8ef854cb9c18ffca46f278f80d369e9ff0b688f97f96a1325c0a0189737b7"} Sep 08 17:03:02.253642 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:02.253604 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:03.088555 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:03.088517 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:03.214713 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:03.214670 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:04.150908 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:04.150875 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb"] Sep 08 17:03:04.155460 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:04.155418 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" Sep 08 17:03:04.159147 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:04.159118 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2746ad-serving-cert\"" Sep 08 17:03:04.169656 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:04.169617 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb"] Sep 08 17:03:04.190806 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:04.190784 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90160bdd-1ef5-43d5-9559-34574b2a69dd-maas-api-tls\") pod \"maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb\" (UID: \"90160bdd-1ef5-43d5-9559-34574b2a69dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" Sep 08 17:03:04.190935 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:04.190823 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90160bdd-1ef5-43d5-9559-34574b2a69dd-metrics-certs\") pod \"maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb\" (UID: \"90160bdd-1ef5-43d5-9559-34574b2a69dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" Sep 08 17:03:04.190935 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:04.190847 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-462jp\" (UniqueName: \"kubernetes.io/projected/90160bdd-1ef5-43d5-9559-34574b2a69dd-kube-api-access-462jp\") pod \"maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb\" (UID: \"90160bdd-1ef5-43d5-9559-34574b2a69dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" Sep 08 17:03:04.190935 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:04.190922 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90160bdd-1ef5-43d5-9559-34574b2a69dd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb\" (UID: \"90160bdd-1ef5-43d5-9559-34574b2a69dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" Sep 08 17:03:04.291859 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:04.291830 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90160bdd-1ef5-43d5-9559-34574b2a69dd-maas-api-tls\") pod \"maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb\" (UID: \"90160bdd-1ef5-43d5-9559-34574b2a69dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" Sep 08 17:03:04.292013 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:04.291868 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90160bdd-1ef5-43d5-9559-34574b2a69dd-metrics-certs\") pod \"maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb\" (UID: \"90160bdd-1ef5-43d5-9559-34574b2a69dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" Sep 08 17:03:04.292013 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:04.291889 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-462jp\" (UniqueName: \"kubernetes.io/projected/90160bdd-1ef5-43d5-9559-34574b2a69dd-kube-api-access-462jp\") pod \"maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb\" (UID: \"90160bdd-1ef5-43d5-9559-34574b2a69dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" Sep 08 17:03:04.292013 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:04.291919 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90160bdd-1ef5-43d5-9559-34574b2a69dd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb\" (UID: \"90160bdd-1ef5-43d5-9559-34574b2a69dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" Sep 08 17:03:04.292555 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:04.292532 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90160bdd-1ef5-43d5-9559-34574b2a69dd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb\" (UID: \"90160bdd-1ef5-43d5-9559-34574b2a69dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" Sep 08 17:03:04.294287 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:04.294264 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90160bdd-1ef5-43d5-9559-34574b2a69dd-metrics-certs\") pod \"maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb\" (UID: \"90160bdd-1ef5-43d5-9559-34574b2a69dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" Sep 08 17:03:04.294474 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:04.294457 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90160bdd-1ef5-43d5-9559-34574b2a69dd-maas-api-tls\") pod \"maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb\" (UID: \"90160bdd-1ef5-43d5-9559-34574b2a69dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" Sep 08 17:03:04.336089 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:04.336065 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-462jp\" (UniqueName: \"kubernetes.io/projected/90160bdd-1ef5-43d5-9559-34574b2a69dd-kube-api-access-462jp\") pod \"maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb\" (UID: \"90160bdd-1ef5-43d5-9559-34574b2a69dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" Sep 08 17:03:04.357634 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:04.357605 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:04.469384 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:04.469361 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" Sep 08 17:03:04.537489 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:04.537062 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:04.751998 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:04.751975 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb"] Sep 08 17:03:04.754077 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:03:04.754050 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90160bdd_1ef5_43d5_9559_34574b2a69dd.slice/crio-16dee8ac97851c127f017cd2db9d3cc816680df228584aeef8f29b1fc0af847d WatchSource:0}: Error finding container 16dee8ac97851c127f017cd2db9d3cc816680df228584aeef8f29b1fc0af847d: Status 404 returned error can't find the container with id 16dee8ac97851c127f017cd2db9d3cc816680df228584aeef8f29b1fc0af847d Sep 08 17:03:05.340096 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:05.340069 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:05.477032 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:05.476998 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:05.749857 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:05.749821 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" event={"ID":"90160bdd-1ef5-43d5-9559-34574b2a69dd","Type":"ContainerStarted","Data":"8354ce8402bd9fd93878b2164f571ccbbf1135ecbd258b0978d568fc067e9692"} Sep 08 17:03:05.750063 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:05.749879 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" event={"ID":"90160bdd-1ef5-43d5-9559-34574b2a69dd","Type":"ContainerStarted","Data":"16dee8ac97851c127f017cd2db9d3cc816680df228584aeef8f29b1fc0af847d"} Sep 08 17:03:05.750063 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:05.749949 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" Sep 08 17:03:05.779644 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:05.779594 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" podStartSLOduration=1.23460623 podStartE2EDuration="1.779579459s" totalImagesPullingTime="544.973229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:03:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:03:05.772639124 +0000 UTC m=+748.509470240" watchObservedRunningTime="2026-09-08 17:03:05.779579459 +0000 UTC m=+748.516410575" Sep 08 17:03:08.107252 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:08.107216 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:08.224153 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:08.224117 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:08.634009 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:08.633963 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:08.844005 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:08.843968 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:08.992860 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:08.992823 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:10.789562 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:10.789526 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7841-9d2b7c" event={"ID":"822c04a4-eae9-48d2-b255-852696fdfdc6","Type":"ContainerStarted","Data":"84da66188d1335bd86d6b0f2b2de3a9693aab1c36cc37c1a0238aa90c55b68a4"} Sep 08 17:03:10.837417 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:10.837370 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7841-9d2b7c" podStartSLOduration=1.868099578 podStartE2EDuration="10.837355576s" totalImagesPullingTime="8.969255998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:03:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:03:10.834861938 +0000 UTC m=+753.571693057" watchObservedRunningTime="2026-09-08 17:03:10.837355576 +0000 UTC m=+753.574186724" Sep 08 17:03:11.305618 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:11.305577 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:11.485192 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:11.485158 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:11.692010 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:11.691976 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7841-9d2b7c"] Sep 08 17:03:11.705115 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:11.705085 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7841-9d2b7c"] Sep 08 17:03:11.762649 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:11.762626 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" Sep 08 17:03:11.801515 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:03:11.801482 2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7841-9d2b7c" container="test-curl-7841-9d2b7c" Sep 08 17:03:11.849710 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:11.849678 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7841-1b36fd"] Sep 08 17:03:11.850317 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:11.850298 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="822c04a4-eae9-48d2-b255-852696fdfdc6" containerName="test-curl-7841-9d2b7c" Sep 08 17:03:11.850371 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:11.850321 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="822c04a4-eae9-48d2-b255-852696fdfdc6" containerName="test-curl-7841-9d2b7c" Sep 08 17:03:11.850465 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:11.850456 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="822c04a4-eae9-48d2-b255-852696fdfdc6" containerName="test-curl-7841-9d2b7c" Sep 08 17:03:11.853954 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:11.853938 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7841-1b36fd" Sep 08 17:03:11.874858 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:11.874838 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7841-1b36fd"] Sep 08 17:03:11.957988 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:11.957911 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g5fh\" (UniqueName: \"kubernetes.io/projected/eb4a05b0-14ef-4e91-960f-487e716adb75-kube-api-access-7g5fh\") pod \"test-curl-7841-1b36fd\" (UID: \"eb4a05b0-14ef-4e91-960f-487e716adb75\") " pod="opendatahub/test-curl-7841-1b36fd" Sep 08 17:03:12.058879 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.058841 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g5fh\" (UniqueName: \"kubernetes.io/projected/eb4a05b0-14ef-4e91-960f-487e716adb75-kube-api-access-7g5fh\") pod \"test-curl-7841-1b36fd\" (UID: \"eb4a05b0-14ef-4e91-960f-487e716adb75\") " pod="opendatahub/test-curl-7841-1b36fd" Sep 08 17:03:12.069085 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.069058 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g5fh\" (UniqueName: \"kubernetes.io/projected/eb4a05b0-14ef-4e91-960f-487e716adb75-kube-api-access-7g5fh\") pod \"test-curl-7841-1b36fd\" (UID: \"eb4a05b0-14ef-4e91-960f-487e716adb75\") " pod="opendatahub/test-curl-7841-1b36fd" Sep 08 17:03:12.162864 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.162760 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7841-1b36fd" Sep 08 17:03:12.428962 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.428929 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc"] Sep 08 17:03:12.433927 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.433905 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" Sep 08 17:03:12.437641 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.437618 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a57edc2e-serving-cert\"" Sep 08 17:03:12.462259 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.462202 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-maas-api-tls\") pod \"maas-api-e2e-ait-a57edc2e-854c564b56-47gcc\" (UID: \"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" Sep 08 17:03:12.462259 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.462231 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-metrics-certs\") pod \"maas-api-e2e-ait-a57edc2e-854c564b56-47gcc\" (UID: \"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" Sep 08 17:03:12.462409 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.462267 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh568\" (UniqueName: \"kubernetes.io/projected/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-kube-api-access-mh568\") pod \"maas-api-e2e-ait-a57edc2e-854c564b56-47gcc\" (UID: \"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" Sep 08 17:03:12.462409 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.462335 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-openshift-service-ca\") pod \"maas-api-e2e-ait-a57edc2e-854c564b56-47gcc\" (UID: \"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" Sep 08 17:03:12.489889 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.489784 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc"] Sep 08 17:03:12.491160 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:03:12.490783 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb4a05b0_14ef_4e91_960f_487e716adb75.slice/crio-29d2d251d86830caac9f1f980617665f1294dab7e1f2f84afcc5b9bce4881756 WatchSource:0}: Error finding container 29d2d251d86830caac9f1f980617665f1294dab7e1f2f84afcc5b9bce4881756: Status 404 returned error can't find the container with id 29d2d251d86830caac9f1f980617665f1294dab7e1f2f84afcc5b9bce4881756 Sep 08 17:03:12.493466 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.493444 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7841-1b36fd"] Sep 08 17:03:12.533087 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.533049 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:12.563798 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.563774 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-maas-api-tls\") pod \"maas-api-e2e-ait-a57edc2e-854c564b56-47gcc\" (UID: \"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" Sep 08 17:03:12.563897 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.563805 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-metrics-certs\") pod \"maas-api-e2e-ait-a57edc2e-854c564b56-47gcc\" (UID: \"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" Sep 08 17:03:12.563897 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.563839 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh568\" (UniqueName: \"kubernetes.io/projected/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-kube-api-access-mh568\") pod \"maas-api-e2e-ait-a57edc2e-854c564b56-47gcc\" (UID: \"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" Sep 08 17:03:12.563897 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.563869 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-openshift-service-ca\") pod \"maas-api-e2e-ait-a57edc2e-854c564b56-47gcc\" (UID: \"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" Sep 08 17:03:12.564504 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.564482 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-openshift-service-ca\") pod \"maas-api-e2e-ait-a57edc2e-854c564b56-47gcc\" (UID: \"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" Sep 08 17:03:12.565998 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.565980 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-maas-api-tls\") pod \"maas-api-e2e-ait-a57edc2e-854c564b56-47gcc\" (UID: \"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" Sep 08 17:03:12.566105 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.566088 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-metrics-certs\") pod \"maas-api-e2e-ait-a57edc2e-854c564b56-47gcc\" (UID: \"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" Sep 08 17:03:12.578687 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.578664 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh568\" (UniqueName: \"kubernetes.io/projected/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-kube-api-access-mh568\") pod \"maas-api-e2e-ait-a57edc2e-854c564b56-47gcc\" (UID: \"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" Sep 08 17:03:12.650853 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.650827 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:12.745757 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.745694 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" Sep 08 17:03:12.813774 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.813538 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7841-9d2b7c" podUID="822c04a4-eae9-48d2-b255-852696fdfdc6" containerName="test-curl-7841-9d2b7c" containerID="cri-o://84da66188d1335bd86d6b0f2b2de3a9693aab1c36cc37c1a0238aa90c55b68a4" gracePeriod=2 Sep 08 17:03:12.813774 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.813694 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7841-1b36fd" event={"ID":"eb4a05b0-14ef-4e91-960f-487e716adb75","Type":"ContainerStarted","Data":"858fef4b7d22bfb0e01208c5e58268b05eda8eef7b8d981dbed3121510f4be43"} Sep 08 17:03:12.813774 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.813731 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7841-1b36fd" event={"ID":"eb4a05b0-14ef-4e91-960f-487e716adb75","Type":"ContainerStarted","Data":"29d2d251d86830caac9f1f980617665f1294dab7e1f2f84afcc5b9bce4881756"} Sep 08 17:03:12.841072 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:12.841028 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7841-1b36fd" podStartSLOduration=1.841012686 podStartE2EDuration="1.841012686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:03:12.838703483 +0000 UTC m=+755.575534599" watchObservedRunningTime="2026-09-08 17:03:12.841012686 +0000 UTC m=+755.577843801" Sep 08 17:03:13.037253 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:13.037224 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc"] Sep 08 17:03:13.039031 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:03:13.039005 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4bdf21_17e4_4202_a0b0_3ddc2af88f87.slice/crio-407a0d65033cd2669c78b88c80788b7b0f79c8d3f493d0bf013a6e7a18c083ae WatchSource:0}: Error finding container 407a0d65033cd2669c78b88c80788b7b0f79c8d3f493d0bf013a6e7a18c083ae: Status 404 returned error can't find the container with id 407a0d65033cd2669c78b88c80788b7b0f79c8d3f493d0bf013a6e7a18c083ae Sep 08 17:03:13.254932 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:13.254898 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:13.391883 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:13.391845 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:13.688732 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:13.688701 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7841-1b36fd"] Sep 08 17:03:13.700043 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:13.700015 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7841-1b36fd"] Sep 08 17:03:13.820096 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:13.820063 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" event={"ID":"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87","Type":"ContainerStarted","Data":"407a0d65033cd2669c78b88c80788b7b0f79c8d3f493d0bf013a6e7a18c083ae"} Sep 08 17:03:13.820500 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:03:13.820319 2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7841-1b36fd" container="test-curl-7841-1b36fd" Sep 08 17:03:14.237205 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:14.237177 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:14.592040 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:14.591953 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:14.829969 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:14.829936 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" event={"ID":"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87","Type":"ContainerStarted","Data":"cc181ef168e8083d28175779a02fad393552b736e9fc0979937de0e62a6bc3f7"} Sep 08 17:03:14.830379 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:14.830001 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7841-1b36fd" podUID="eb4a05b0-14ef-4e91-960f-487e716adb75" containerName="test-curl-7841-1b36fd" containerID="cri-o://858fef4b7d22bfb0e01208c5e58268b05eda8eef7b8d981dbed3121510f4be43" gracePeriod=2 Sep 08 17:03:14.830379 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:14.830025 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" Sep 08 17:03:14.861498 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:14.861454 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" podStartSLOduration=2.0299706 podStartE2EDuration="2.861438086s" totalImagesPullingTime="831.467486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:03:14.85898085 +0000 UTC m=+757.595811967" watchObservedRunningTime="2026-09-08 17:03:14.861438086 +0000 UTC m=+757.598269192" Sep 08 17:03:14.996063 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:14.996041 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7841-9d2b7c" Sep 08 17:03:15.087780 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:15.087746 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtndg\" (UniqueName: \"kubernetes.io/projected/822c04a4-eae9-48d2-b255-852696fdfdc6-kube-api-access-mtndg\") pod \"822c04a4-eae9-48d2-b255-852696fdfdc6\" (UID: \"822c04a4-eae9-48d2-b255-852696fdfdc6\") " Sep 08 17:03:15.089774 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:15.089748 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822c04a4-eae9-48d2-b255-852696fdfdc6-kube-api-access-mtndg" (OuterVolumeSpecName: "kube-api-access-mtndg") pod "822c04a4-eae9-48d2-b255-852696fdfdc6" (UID: "822c04a4-eae9-48d2-b255-852696fdfdc6"). InnerVolumeSpecName "kube-api-access-mtndg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:03:15.188814 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:15.188787 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtndg\" (UniqueName: \"kubernetes.io/projected/822c04a4-eae9-48d2-b255-852696fdfdc6-kube-api-access-mtndg\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:03:15.398031 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:15.397995 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:15.761380 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:15.761345 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="822c04a4-eae9-48d2-b255-852696fdfdc6" path="/var/lib/kubelet/pods/822c04a4-eae9-48d2-b255-852696fdfdc6/volumes" Sep 08 17:03:15.837280 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:15.837243 2565 generic.go:356] "Generic (PLEG): container finished" podID="822c04a4-eae9-48d2-b255-852696fdfdc6" containerID="84da66188d1335bd86d6b0f2b2de3a9693aab1c36cc37c1a0238aa90c55b68a4" exitCode=137 Sep 08 17:03:15.837666 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:15.837288 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7841-9d2b7c" Sep 08 17:03:15.837666 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:15.837345 2565 scope.go:117] "RemoveContainer" containerID="84da66188d1335bd86d6b0f2b2de3a9693aab1c36cc37c1a0238aa90c55b68a4" Sep 08 17:03:15.846562 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:15.846535 2565 scope.go:117] "RemoveContainer" containerID="84da66188d1335bd86d6b0f2b2de3a9693aab1c36cc37c1a0238aa90c55b68a4" Sep 08 17:03:15.846943 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:03:15.846923 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84da66188d1335bd86d6b0f2b2de3a9693aab1c36cc37c1a0238aa90c55b68a4\": container with ID starting with 84da66188d1335bd86d6b0f2b2de3a9693aab1c36cc37c1a0238aa90c55b68a4 not found: ID does not exist" containerID="84da66188d1335bd86d6b0f2b2de3a9693aab1c36cc37c1a0238aa90c55b68a4" Sep 08 17:03:15.847049 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:15.846955 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84da66188d1335bd86d6b0f2b2de3a9693aab1c36cc37c1a0238aa90c55b68a4"} err="failed to get container status \"84da66188d1335bd86d6b0f2b2de3a9693aab1c36cc37c1a0238aa90c55b68a4\": rpc error: code = NotFound desc = could not find container \"84da66188d1335bd86d6b0f2b2de3a9693aab1c36cc37c1a0238aa90c55b68a4\": container with ID starting with 84da66188d1335bd86d6b0f2b2de3a9693aab1c36cc37c1a0238aa90c55b68a4 not found: ID does not exist" Sep 08 17:03:17.012839 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:17.012815 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7841-1b36fd" Sep 08 17:03:17.106599 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:17.106533 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7g5fh\" (UniqueName: \"kubernetes.io/projected/eb4a05b0-14ef-4e91-960f-487e716adb75-kube-api-access-7g5fh\") pod \"eb4a05b0-14ef-4e91-960f-487e716adb75\" (UID: \"eb4a05b0-14ef-4e91-960f-487e716adb75\") " Sep 08 17:03:17.108550 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:17.108523 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb4a05b0-14ef-4e91-960f-487e716adb75-kube-api-access-7g5fh" (OuterVolumeSpecName: "kube-api-access-7g5fh") pod "eb4a05b0-14ef-4e91-960f-487e716adb75" (UID: "eb4a05b0-14ef-4e91-960f-487e716adb75"). InnerVolumeSpecName "kube-api-access-7g5fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:03:17.207339 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:17.207305 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g5fh\" (UniqueName: \"kubernetes.io/projected/eb4a05b0-14ef-4e91-960f-487e716adb75-kube-api-access-7g5fh\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:03:17.761668 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:17.761635 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb4a05b0-14ef-4e91-960f-487e716adb75" path="/var/lib/kubelet/pods/eb4a05b0-14ef-4e91-960f-487e716adb75/volumes" Sep 08 17:03:17.852272 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:17.852240 2565 generic.go:356] "Generic (PLEG): container finished" podID="eb4a05b0-14ef-4e91-960f-487e716adb75" containerID="858fef4b7d22bfb0e01208c5e58268b05eda8eef7b8d981dbed3121510f4be43" exitCode=137 Sep 08 17:03:17.852566 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:17.852288 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7841-1b36fd" Sep 08 17:03:17.852566 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:17.852345 2565 scope.go:117] "RemoveContainer" containerID="858fef4b7d22bfb0e01208c5e58268b05eda8eef7b8d981dbed3121510f4be43" Sep 08 17:03:17.863065 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:17.863034 2565 scope.go:117] "RemoveContainer" containerID="858fef4b7d22bfb0e01208c5e58268b05eda8eef7b8d981dbed3121510f4be43" Sep 08 17:03:17.863439 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:03:17.863396 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"858fef4b7d22bfb0e01208c5e58268b05eda8eef7b8d981dbed3121510f4be43\": container with ID starting with 858fef4b7d22bfb0e01208c5e58268b05eda8eef7b8d981dbed3121510f4be43 not found: ID does not exist" containerID="858fef4b7d22bfb0e01208c5e58268b05eda8eef7b8d981dbed3121510f4be43" Sep 08 17:03:17.863548 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:17.863458 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858fef4b7d22bfb0e01208c5e58268b05eda8eef7b8d981dbed3121510f4be43"} err="failed to get container status \"858fef4b7d22bfb0e01208c5e58268b05eda8eef7b8d981dbed3121510f4be43\": rpc error: code = NotFound desc = could not find container \"858fef4b7d22bfb0e01208c5e58268b05eda8eef7b8d981dbed3121510f4be43\": container with ID starting with 858fef4b7d22bfb0e01208c5e58268b05eda8eef7b8d981dbed3121510f4be43 not found: ID does not exist" Sep 08 17:03:17.875257 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:17.875231 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:18.044611 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:18.044532 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:18.226723 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:18.226688 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:18.491986 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:18.491950 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:18.763582 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:18.763500 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:19.264816 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:19.264773 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx"] Sep 08 17:03:19.265515 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:19.265485 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb4a05b0-14ef-4e91-960f-487e716adb75" containerName="test-curl-7841-1b36fd" Sep 08 17:03:19.265515 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:19.265514 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb4a05b0-14ef-4e91-960f-487e716adb75" containerName="test-curl-7841-1b36fd" Sep 08 17:03:19.265744 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:19.265622 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb4a05b0-14ef-4e91-960f-487e716adb75" containerName="test-curl-7841-1b36fd" Sep 08 17:03:19.278812 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:19.278783 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" Sep 08 17:03:19.281609 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:19.281582 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-66e7a8-serving-cert\"" Sep 08 17:03:19.294292 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:19.294268 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx"] Sep 08 17:03:19.326105 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:19.326069 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-openshift-service-ca\") pod \"maas-api-e2e-autores-66e7a8-67d5575d48-89tmx\" (UID: \"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" Sep 08 17:03:19.326233 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:19.326175 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-metrics-certs\") pod \"maas-api-e2e-autores-66e7a8-67d5575d48-89tmx\" (UID: \"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" Sep 08 17:03:19.326308 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:19.326232 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smg66\" (UniqueName: \"kubernetes.io/projected/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-kube-api-access-smg66\") pod \"maas-api-e2e-autores-66e7a8-67d5575d48-89tmx\" (UID: \"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" Sep 08 17:03:19.326308 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:19.326274 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-maas-api-tls\") pod \"maas-api-e2e-autores-66e7a8-67d5575d48-89tmx\" (UID: \"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" Sep 08 17:03:19.427504 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:19.427468 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-openshift-service-ca\") pod \"maas-api-e2e-autores-66e7a8-67d5575d48-89tmx\" (UID: \"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" Sep 08 17:03:19.427691 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:19.427547 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-metrics-certs\") pod \"maas-api-e2e-autores-66e7a8-67d5575d48-89tmx\" (UID: \"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" Sep 08 17:03:19.427691 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:19.427592 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smg66\" (UniqueName: \"kubernetes.io/projected/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-kube-api-access-smg66\") pod \"maas-api-e2e-autores-66e7a8-67d5575d48-89tmx\" (UID: \"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" Sep 08 17:03:19.427691 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:19.427633 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-maas-api-tls\") pod \"maas-api-e2e-autores-66e7a8-67d5575d48-89tmx\" (UID: \"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" Sep 08 17:03:19.427879 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:03:19.427787 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-serving-cert: secret "maas-api-e2e-autores-66e7a8-serving-cert" not found Sep 08 17:03:19.427879 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:03:19.427859 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-maas-api-tls podName:a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc nodeName:}" failed. No retries permitted until 2026-09-08 17:03:19.927837787 +0000 UTC m=+762.664668881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-maas-api-tls") pod "maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" (UID: "a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc") : secret "maas-api-e2e-autores-66e7a8-serving-cert" not found Sep 08 17:03:19.428303 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:19.428278 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-openshift-service-ca\") pod \"maas-api-e2e-autores-66e7a8-67d5575d48-89tmx\" (UID: \"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" Sep 08 17:03:19.430184 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:19.430159 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-metrics-certs\") pod \"maas-api-e2e-autores-66e7a8-67d5575d48-89tmx\" (UID: \"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" Sep 08 17:03:19.443723 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:19.443698 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smg66\" (UniqueName: \"kubernetes.io/projected/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-kube-api-access-smg66\") pod \"maas-api-e2e-autores-66e7a8-67d5575d48-89tmx\" (UID: \"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" Sep 08 17:03:19.698322 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:19.698286 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:19.869554 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:19.869495 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:19.934519 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:19.934484 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-maas-api-tls\") pod \"maas-api-e2e-autores-66e7a8-67d5575d48-89tmx\" (UID: \"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" Sep 08 17:03:19.936925 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:19.936898 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-maas-api-tls\") pod \"maas-api-e2e-autores-66e7a8-67d5575d48-89tmx\" (UID: \"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" Sep 08 17:03:20.028598 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:20.028518 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:20.194936 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:20.194902 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" Sep 08 17:03:20.224810 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:20.224770 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:20.506685 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:20.506656 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx"] Sep 08 17:03:20.508916 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:03:20.508887 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83d7a8b_b0b2_44e6_87b4_dc8d105ab7cc.slice/crio-b9c8fffc68a03664624c5656c74fd2d0fca8961f23fc6a83d4326669a6c0f4b4 WatchSource:0}: Error finding container b9c8fffc68a03664624c5656c74fd2d0fca8961f23fc6a83d4326669a6c0f4b4: Status 404 returned error can't find the container with id b9c8fffc68a03664624c5656c74fd2d0fca8961f23fc6a83d4326669a6c0f4b4 Sep 08 17:03:20.517867 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:20.517842 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:20.639809 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:20.639777 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:20.842583 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:20.842509 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" Sep 08 17:03:20.874691 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:20.874654 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" event={"ID":"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc","Type":"ContainerStarted","Data":"b9c8fffc68a03664624c5656c74fd2d0fca8961f23fc6a83d4326669a6c0f4b4"} Sep 08 17:03:21.476216 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:21.476177 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:21.884642 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:21.884546 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" event={"ID":"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc","Type":"ContainerStarted","Data":"a5326241c404f717ba354166fc172dfd9b59de39db19576d9680ef5465cf8045"} Sep 08 17:03:21.884642 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:21.884635 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" Sep 08 17:03:21.916367 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:21.916320 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" podStartSLOduration=2.400537972 podStartE2EDuration="2.916305081s" totalImagesPullingTime="515.767109ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:03:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:03:21.91437019 +0000 UTC m=+764.651201306" watchObservedRunningTime="2026-09-08 17:03:21.916305081 +0000 UTC m=+764.653136226" Sep 08 17:03:23.283446 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:23.283401 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:25.878990 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:25.878955 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:27.896672 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:27.896636 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" Sep 08 17:03:28.001223 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:28.001178 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:29.365905 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:29.365876 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:29.538133 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:29.538098 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:30.395809 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:30.395763 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:30.603740 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:30.603696 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:30.891921 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:30.891881 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:31.204532 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:31.204496 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:32.180827 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:32.180789 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:32.411758 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:32.411719 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:33.161544 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:33.161504 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:33.334658 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:33.334618 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj"] Sep 08 17:03:33.338343 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:33.338321 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" Sep 08 17:03:33.342101 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:33.342082 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5a7cab-serving-cert\"" Sep 08 17:03:33.366654 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:33.366633 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj"] Sep 08 17:03:33.410759 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:33.410731 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:33.455063 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:33.455038 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmwbk\" (UniqueName: \"kubernetes.io/projected/73db64db-594c-4f41-b959-645623e61c11-kube-api-access-pmwbk\") pod \"maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj\" (UID: \"73db64db-594c-4f41-b959-645623e61c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" Sep 08 17:03:33.455184 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:33.455160 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73db64db-594c-4f41-b959-645623e61c11-maas-api-tls\") pod \"maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj\" (UID: \"73db64db-594c-4f41-b959-645623e61c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" Sep 08 17:03:33.455250 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:33.455184 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73db64db-594c-4f41-b959-645623e61c11-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj\" (UID: \"73db64db-594c-4f41-b959-645623e61c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" Sep 08 17:03:33.455250 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:33.455219 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73db64db-594c-4f41-b959-645623e61c11-metrics-certs\") pod \"maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj\" (UID: \"73db64db-594c-4f41-b959-645623e61c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" Sep 08 17:03:33.556007 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:33.555980 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73db64db-594c-4f41-b959-645623e61c11-maas-api-tls\") pod \"maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj\" (UID: \"73db64db-594c-4f41-b959-645623e61c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" Sep 08 17:03:33.556161 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:33.556011 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73db64db-594c-4f41-b959-645623e61c11-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj\" (UID: \"73db64db-594c-4f41-b959-645623e61c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" Sep 08 17:03:33.556161 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:33.556038 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73db64db-594c-4f41-b959-645623e61c11-metrics-certs\") pod \"maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj\" (UID: \"73db64db-594c-4f41-b959-645623e61c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" Sep 08 17:03:33.556161 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:33.556075 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmwbk\" (UniqueName: \"kubernetes.io/projected/73db64db-594c-4f41-b959-645623e61c11-kube-api-access-pmwbk\") pod \"maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj\" (UID: \"73db64db-594c-4f41-b959-645623e61c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" Sep 08 17:03:33.556161 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:03:33.556148 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-serving-cert: secret "maas-api-e2e-inf-b-5a7cab-serving-cert" not found Sep 08 17:03:33.556374 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:03:33.556226 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73db64db-594c-4f41-b959-645623e61c11-maas-api-tls podName:73db64db-594c-4f41-b959-645623e61c11 nodeName:}" failed. No retries permitted until 2026-09-08 17:03:34.056202694 +0000 UTC m=+776.793033791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/73db64db-594c-4f41-b959-645623e61c11-maas-api-tls") pod "maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" (UID: "73db64db-594c-4f41-b959-645623e61c11") : secret "maas-api-e2e-inf-b-5a7cab-serving-cert" not found Sep 08 17:03:33.556622 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:33.556600 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73db64db-594c-4f41-b959-645623e61c11-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj\" (UID: \"73db64db-594c-4f41-b959-645623e61c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" Sep 08 17:03:33.558590 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:33.558566 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73db64db-594c-4f41-b959-645623e61c11-metrics-certs\") pod \"maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj\" (UID: \"73db64db-594c-4f41-b959-645623e61c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" Sep 08 17:03:33.580834 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:33.580806 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmwbk\" (UniqueName: \"kubernetes.io/projected/73db64db-594c-4f41-b959-645623e61c11-kube-api-access-pmwbk\") pod \"maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj\" (UID: \"73db64db-594c-4f41-b959-645623e61c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" Sep 08 17:03:33.582918 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:33.582892 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:33.768726 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:33.768642 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:34.061583 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:34.061505 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73db64db-594c-4f41-b959-645623e61c11-maas-api-tls\") pod \"maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj\" (UID: \"73db64db-594c-4f41-b959-645623e61c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" Sep 08 17:03:34.063874 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:34.063853 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73db64db-594c-4f41-b959-645623e61c11-maas-api-tls\") pod \"maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj\" (UID: \"73db64db-594c-4f41-b959-645623e61c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" Sep 08 17:03:34.249811 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:34.249772 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" Sep 08 17:03:34.318190 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:34.318096 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:34.501124 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:34.501085 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:34.559110 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:34.559085 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj"] Sep 08 17:03:34.560942 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:03:34.560910 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73db64db_594c_4f41_b959_645623e61c11.slice/crio-495333cc953ef1ee728558a5bf31a6b1a3efff324699f313a88f09fc6e4b8201 WatchSource:0}: Error finding container 495333cc953ef1ee728558a5bf31a6b1a3efff324699f313a88f09fc6e4b8201: Status 404 returned error can't find the container with id 495333cc953ef1ee728558a5bf31a6b1a3efff324699f313a88f09fc6e4b8201 Sep 08 17:03:34.966739 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:34.966697 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" event={"ID":"73db64db-594c-4f41-b959-645623e61c11","Type":"ContainerStarted","Data":"495333cc953ef1ee728558a5bf31a6b1a3efff324699f313a88f09fc6e4b8201"} Sep 08 17:03:35.219925 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:35.219862 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:35.974063 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:35.974018 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" event={"ID":"73db64db-594c-4f41-b959-645623e61c11","Type":"ContainerStarted","Data":"3640a5bb16a24b4854e95913eda142f89fb990e253edf84d104b107c646d4ce4"} Sep 08 17:03:35.974469 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:35.974156 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" Sep 08 17:03:35.995669 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:35.995622 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" podStartSLOduration=2.349356161 podStartE2EDuration="2.995608417s" totalImagesPullingTime="646.252256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:03:35.994179989 +0000 UTC m=+778.731011105" watchObservedRunningTime="2026-09-08 17:03:35.995608417 +0000 UTC m=+778.732439590" Sep 08 17:03:36.357736 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:36.357650 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:37.377330 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:37.377289 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:38.270528 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:38.270496 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:38.432444 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:38.432400 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:39.526658 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:39.526623 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:41.984979 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:41.984944 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" Sep 08 17:03:44.398800 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:44.398755 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:48.038393 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:48.038357 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:53.609906 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:53.609864 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:53.965419 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:53.965383 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc"] Sep 08 17:03:53.965615 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:53.965594 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" podUID="ec4bdf21-17e4-4202-a0b0-3ddc2af88f87" containerName="maas-api" containerID="cri-o://cc181ef168e8083d28175779a02fad393552b736e9fc0979937de0e62a6bc3f7" gracePeriod=30 Sep 08 17:03:54.044318 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:54.044275 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:54.088482 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:54.088444 2565 generic.go:356] "Generic (PLEG): container finished" podID="ec4bdf21-17e4-4202-a0b0-3ddc2af88f87" containerID="cc181ef168e8083d28175779a02fad393552b736e9fc0979937de0e62a6bc3f7" exitCode=0 Sep 08 17:03:54.088646 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:54.088624 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" event={"ID":"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87","Type":"ContainerDied","Data":"cc181ef168e8083d28175779a02fad393552b736e9fc0979937de0e62a6bc3f7"} Sep 08 17:03:54.263418 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:54.263389 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" Sep 08 17:03:54.339954 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:54.339922 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:54.358217 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:54.358193 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-maas-api-tls\") pod \"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87\" (UID: \"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87\") " Sep 08 17:03:54.358375 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:54.358236 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-openshift-service-ca\") pod \"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87\" (UID: \"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87\") " Sep 08 17:03:54.358375 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:54.358268 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mh568\" (UniqueName: \"kubernetes.io/projected/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-kube-api-access-mh568\") pod \"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87\" (UID: \"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87\") " Sep 08 17:03:54.358375 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:54.358320 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-metrics-certs\") pod \"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87\" (UID: \"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87\") " Sep 08 17:03:54.358706 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:54.358663 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec4bdf21-17e4-4202-a0b0-3ddc2af88f87" (UID: "ec4bdf21-17e4-4202-a0b0-3ddc2af88f87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:03:54.360597 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:54.360546 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-kube-api-access-mh568" (OuterVolumeSpecName: "kube-api-access-mh568") pod "ec4bdf21-17e4-4202-a0b0-3ddc2af88f87" (UID: "ec4bdf21-17e4-4202-a0b0-3ddc2af88f87"). InnerVolumeSpecName "kube-api-access-mh568". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:03:54.360866 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:54.360832 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ec4bdf21-17e4-4202-a0b0-3ddc2af88f87" (UID: "ec4bdf21-17e4-4202-a0b0-3ddc2af88f87"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:03:54.360949 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:54.360906 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec4bdf21-17e4-4202-a0b0-3ddc2af88f87" (UID: "ec4bdf21-17e4-4202-a0b0-3ddc2af88f87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:03:54.459841 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:54.459816 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-openshift-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:03:54.459841 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:54.459838 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh568\" (UniqueName: \"kubernetes.io/projected/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-kube-api-access-mh568\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:03:54.459986 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:54.459850 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:03:54.459986 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:54.459858 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87-maas-api-tls\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:03:54.609630 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:54.609554 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:54.756700 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:54.756672 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:54.974576 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:54.974542 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:55.096229 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:55.096197 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" Sep 08 17:03:55.096229 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:55.096218 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc" event={"ID":"ec4bdf21-17e4-4202-a0b0-3ddc2af88f87","Type":"ContainerDied","Data":"407a0d65033cd2669c78b88c80788b7b0f79c8d3f493d0bf013a6e7a18c083ae"} Sep 08 17:03:55.096460 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:55.096271 2565 scope.go:117] "RemoveContainer" containerID="cc181ef168e8083d28175779a02fad393552b736e9fc0979937de0e62a6bc3f7" Sep 08 17:03:55.131325 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:55.131298 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc"] Sep 08 17:03:55.144375 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:55.144353 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a57edc2e-854c564b56-47gcc"] Sep 08 17:03:55.448492 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:55.448457 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:55.760675 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:55.760601 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec4bdf21-17e4-4202-a0b0-3ddc2af88f87" path="/var/lib/kubelet/pods/ec4bdf21-17e4-4202-a0b0-3ddc2af88f87/volumes" Sep 08 17:03:55.782513 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:55.782464 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:56.063058 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:56.062985 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:56.843882 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:56.843847 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:57.228464 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:57.228411 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:03:59.832276 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:03:59.832233 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:00.004897 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.004860 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:00.115572 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.115470 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs"] Sep 08 17:04:00.115936 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.115922 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec4bdf21-17e4-4202-a0b0-3ddc2af88f87" containerName="maas-api" Sep 08 17:04:00.115983 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.115937 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec4bdf21-17e4-4202-a0b0-3ddc2af88f87" containerName="maas-api" Sep 08 17:04:00.116051 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.116042 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec4bdf21-17e4-4202-a0b0-3ddc2af88f87" containerName="maas-api" Sep 08 17:04:00.120759 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.120735 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" Sep 08 17:04:00.123394 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.123365 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-56569e47-serving-cert\"" Sep 08 17:04:00.153176 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.153138 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs"] Sep 08 17:04:00.207770 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.207741 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/482a77b1-677c-4bc4-a268-7f130e57d6a3-maas-api-tls\") pod \"maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs\" (UID: \"482a77b1-677c-4bc4-a268-7f130e57d6a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" Sep 08 17:04:00.207898 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.207789 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482a77b1-677c-4bc4-a268-7f130e57d6a3-metrics-certs\") pod \"maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs\" (UID: \"482a77b1-677c-4bc4-a268-7f130e57d6a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" Sep 08 17:04:00.207898 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.207810 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc2nx\" (UniqueName: \"kubernetes.io/projected/482a77b1-677c-4bc4-a268-7f130e57d6a3-kube-api-access-zc2nx\") pod \"maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs\" (UID: \"482a77b1-677c-4bc4-a268-7f130e57d6a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" Sep 08 17:04:00.207898 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.207859 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/482a77b1-677c-4bc4-a268-7f130e57d6a3-openshift-service-ca\") pod \"maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs\" (UID: \"482a77b1-677c-4bc4-a268-7f130e57d6a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" Sep 08 17:04:00.247080 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.247044 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:00.308609 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.308577 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482a77b1-677c-4bc4-a268-7f130e57d6a3-metrics-certs\") pod \"maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs\" (UID: \"482a77b1-677c-4bc4-a268-7f130e57d6a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" Sep 08 17:04:00.308609 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.308612 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc2nx\" (UniqueName: \"kubernetes.io/projected/482a77b1-677c-4bc4-a268-7f130e57d6a3-kube-api-access-zc2nx\") pod \"maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs\" (UID: \"482a77b1-677c-4bc4-a268-7f130e57d6a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" Sep 08 17:04:00.308853 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.308650 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/482a77b1-677c-4bc4-a268-7f130e57d6a3-openshift-service-ca\") pod \"maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs\" (UID: \"482a77b1-677c-4bc4-a268-7f130e57d6a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" Sep 08 17:04:00.308853 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.308711 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/482a77b1-677c-4bc4-a268-7f130e57d6a3-maas-api-tls\") pod \"maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs\" (UID: \"482a77b1-677c-4bc4-a268-7f130e57d6a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" Sep 08 17:04:00.308853 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:04:00.308813 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-serving-cert: secret "maas-api-e2e-migrate-56569e47-serving-cert" not found Sep 08 17:04:00.309012 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:04:00.308922 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/482a77b1-677c-4bc4-a268-7f130e57d6a3-maas-api-tls podName:482a77b1-677c-4bc4-a268-7f130e57d6a3 nodeName:}" failed. No retries permitted until 2026-09-08 17:04:00.808894476 +0000 UTC m=+803.545725570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/482a77b1-677c-4bc4-a268-7f130e57d6a3-maas-api-tls") pod "maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" (UID: "482a77b1-677c-4bc4-a268-7f130e57d6a3") : secret "maas-api-e2e-migrate-56569e47-serving-cert" not found Sep 08 17:04:00.309332 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.309314 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/482a77b1-677c-4bc4-a268-7f130e57d6a3-openshift-service-ca\") pod \"maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs\" (UID: \"482a77b1-677c-4bc4-a268-7f130e57d6a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" Sep 08 17:04:00.310963 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.310941 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482a77b1-677c-4bc4-a268-7f130e57d6a3-metrics-certs\") pod \"maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs\" (UID: \"482a77b1-677c-4bc4-a268-7f130e57d6a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" Sep 08 17:04:00.331794 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.331764 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc2nx\" (UniqueName: \"kubernetes.io/projected/482a77b1-677c-4bc4-a268-7f130e57d6a3-kube-api-access-zc2nx\") pod \"maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs\" (UID: \"482a77b1-677c-4bc4-a268-7f130e57d6a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" Sep 08 17:04:00.438366 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.438329 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:00.565550 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.565514 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:00.813140 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.813056 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/482a77b1-677c-4bc4-a268-7f130e57d6a3-maas-api-tls\") pod \"maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs\" (UID: \"482a77b1-677c-4bc4-a268-7f130e57d6a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" Sep 08 17:04:00.815566 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:00.815544 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/482a77b1-677c-4bc4-a268-7f130e57d6a3-maas-api-tls\") pod \"maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs\" (UID: \"482a77b1-677c-4bc4-a268-7f130e57d6a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" Sep 08 17:04:01.003551 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:01.003504 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:01.032605 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:01.032576 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" Sep 08 17:04:01.164493 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:01.164449 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:01.354258 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:04:01.354224 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482a77b1_677c_4bc4_a268_7f130e57d6a3.slice/crio-eeac69ec5aa9b96d8c88c5ff5a65cfd0f9cadbe6990f03da9cc814124f28c1ca WatchSource:0}: Error finding container eeac69ec5aa9b96d8c88c5ff5a65cfd0f9cadbe6990f03da9cc814124f28c1ca: Status 404 returned error can't find the container with id eeac69ec5aa9b96d8c88c5ff5a65cfd0f9cadbe6990f03da9cc814124f28c1ca Sep 08 17:04:01.355517 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:01.355490 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs"] Sep 08 17:04:02.141234 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:02.141138 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" event={"ID":"482a77b1-677c-4bc4-a268-7f130e57d6a3","Type":"ContainerStarted","Data":"816eec31505e133371bb9c2bc893a6b91084a8900f4d2cd508b29bf6c4251915"} Sep 08 17:04:02.141234 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:02.141189 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" event={"ID":"482a77b1-677c-4bc4-a268-7f130e57d6a3","Type":"ContainerStarted","Data":"eeac69ec5aa9b96d8c88c5ff5a65cfd0f9cadbe6990f03da9cc814124f28c1ca"} Sep 08 17:04:02.141708 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:02.141338 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" Sep 08 17:04:02.194895 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:02.194840 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" podStartSLOduration=1.7710441430000001 podStartE2EDuration="2.194823004s" totalImagesPullingTime="423.778861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:04:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:04:02.192656931 +0000 UTC m=+804.929488048" watchObservedRunningTime="2026-09-08 17:04:02.194823004 +0000 UTC m=+804.931654140" Sep 08 17:04:02.357185 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:02.357147 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:08.154154 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:08.154122 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" Sep 08 17:04:31.258018 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:31.257971 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:31.517828 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:31.517697 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs"] Sep 08 17:04:31.518055 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:31.518003 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" podUID="482a77b1-677c-4bc4-a268-7f130e57d6a3" containerName="maas-api" containerID="cri-o://816eec31505e133371bb9c2bc893a6b91084a8900f4d2cd508b29bf6c4251915" gracePeriod=30 Sep 08 17:04:31.562003 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:31.561969 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:31.661237 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:31.661200 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:31.794049 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:31.794026 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" Sep 08 17:04:31.894762 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:31.894725 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482a77b1-677c-4bc4-a268-7f130e57d6a3-metrics-certs\") pod \"482a77b1-677c-4bc4-a268-7f130e57d6a3\" (UID: \"482a77b1-677c-4bc4-a268-7f130e57d6a3\") " Sep 08 17:04:31.894946 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:31.894792 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/482a77b1-677c-4bc4-a268-7f130e57d6a3-maas-api-tls\") pod \"482a77b1-677c-4bc4-a268-7f130e57d6a3\" (UID: \"482a77b1-677c-4bc4-a268-7f130e57d6a3\") " Sep 08 17:04:31.894946 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:31.894831 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/482a77b1-677c-4bc4-a268-7f130e57d6a3-openshift-service-ca\") pod \"482a77b1-677c-4bc4-a268-7f130e57d6a3\" (UID: \"482a77b1-677c-4bc4-a268-7f130e57d6a3\") " Sep 08 17:04:31.894946 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:31.894857 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zc2nx\" (UniqueName: \"kubernetes.io/projected/482a77b1-677c-4bc4-a268-7f130e57d6a3-kube-api-access-zc2nx\") pod \"482a77b1-677c-4bc4-a268-7f130e57d6a3\" (UID: \"482a77b1-677c-4bc4-a268-7f130e57d6a3\") " Sep 08 17:04:31.895215 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:31.895188 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/482a77b1-677c-4bc4-a268-7f130e57d6a3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "482a77b1-677c-4bc4-a268-7f130e57d6a3" (UID: "482a77b1-677c-4bc4-a268-7f130e57d6a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:04:31.896998 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:31.896942 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/482a77b1-677c-4bc4-a268-7f130e57d6a3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "482a77b1-677c-4bc4-a268-7f130e57d6a3" (UID: "482a77b1-677c-4bc4-a268-7f130e57d6a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:04:31.897107 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:31.897018 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/482a77b1-677c-4bc4-a268-7f130e57d6a3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "482a77b1-677c-4bc4-a268-7f130e57d6a3" (UID: "482a77b1-677c-4bc4-a268-7f130e57d6a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:04:31.897107 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:31.897039 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482a77b1-677c-4bc4-a268-7f130e57d6a3-kube-api-access-zc2nx" (OuterVolumeSpecName: "kube-api-access-zc2nx") pod "482a77b1-677c-4bc4-a268-7f130e57d6a3" (UID: "482a77b1-677c-4bc4-a268-7f130e57d6a3"). InnerVolumeSpecName "kube-api-access-zc2nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:04:31.995640 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:31.995604 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/482a77b1-677c-4bc4-a268-7f130e57d6a3-maas-api-tls\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:04:31.995640 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:31.995642 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/482a77b1-677c-4bc4-a268-7f130e57d6a3-openshift-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:04:31.995824 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:31.995658 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc2nx\" (UniqueName: \"kubernetes.io/projected/482a77b1-677c-4bc4-a268-7f130e57d6a3-kube-api-access-zc2nx\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:04:31.995824 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:31.995672 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482a77b1-677c-4bc4-a268-7f130e57d6a3-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:04:32.042338 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:32.042247 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:32.155883 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:32.155848 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:32.328692 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:32.328605 2565 generic.go:356] "Generic (PLEG): container finished" podID="482a77b1-677c-4bc4-a268-7f130e57d6a3" containerID="816eec31505e133371bb9c2bc893a6b91084a8900f4d2cd508b29bf6c4251915" exitCode=0 Sep 08 17:04:32.328692 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:32.328645 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" Sep 08 17:04:32.329154 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:32.328688 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" event={"ID":"482a77b1-677c-4bc4-a268-7f130e57d6a3","Type":"ContainerDied","Data":"816eec31505e133371bb9c2bc893a6b91084a8900f4d2cd508b29bf6c4251915"} Sep 08 17:04:32.329154 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:32.328740 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs" event={"ID":"482a77b1-677c-4bc4-a268-7f130e57d6a3","Type":"ContainerDied","Data":"eeac69ec5aa9b96d8c88c5ff5a65cfd0f9cadbe6990f03da9cc814124f28c1ca"} Sep 08 17:04:32.329154 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:32.328764 2565 scope.go:117] "RemoveContainer" containerID="816eec31505e133371bb9c2bc893a6b91084a8900f4d2cd508b29bf6c4251915" Sep 08 17:04:32.339060 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:32.339043 2565 scope.go:117] "RemoveContainer" containerID="816eec31505e133371bb9c2bc893a6b91084a8900f4d2cd508b29bf6c4251915" Sep 08 17:04:32.339387 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:04:32.339359 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"816eec31505e133371bb9c2bc893a6b91084a8900f4d2cd508b29bf6c4251915\": container with ID starting with 816eec31505e133371bb9c2bc893a6b91084a8900f4d2cd508b29bf6c4251915 not found: ID does not exist" containerID="816eec31505e133371bb9c2bc893a6b91084a8900f4d2cd508b29bf6c4251915" Sep 08 17:04:32.339513 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:32.339396 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"816eec31505e133371bb9c2bc893a6b91084a8900f4d2cd508b29bf6c4251915"} err="failed to get container status \"816eec31505e133371bb9c2bc893a6b91084a8900f4d2cd508b29bf6c4251915\": rpc error: code = NotFound desc = could not find container \"816eec31505e133371bb9c2bc893a6b91084a8900f4d2cd508b29bf6c4251915\": container with ID starting with 816eec31505e133371bb9c2bc893a6b91084a8900f4d2cd508b29bf6c4251915 not found: ID does not exist" Sep 08 17:04:32.357699 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:32.357677 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs"] Sep 08 17:04:32.361701 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:32.361671 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-56569e47-79cd98b9f8-d5dxs"] Sep 08 17:04:32.443377 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:32.443347 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:32.781227 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:32.781196 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:33.054912 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:33.054809 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:33.760346 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:33.760314 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="482a77b1-677c-4bc4-a268-7f130e57d6a3" path="/var/lib/kubelet/pods/482a77b1-677c-4bc4-a268-7f130e57d6a3/volumes" Sep 08 17:04:35.824647 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:35.824603 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:35.997203 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:35.997169 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:36.909457 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:36.909405 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9"] Sep 08 17:04:36.909877 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:36.909862 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="482a77b1-677c-4bc4-a268-7f130e57d6a3" containerName="maas-api" Sep 08 17:04:36.909927 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:36.909881 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="482a77b1-677c-4bc4-a268-7f130e57d6a3" containerName="maas-api" Sep 08 17:04:36.909979 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:36.909969 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="482a77b1-677c-4bc4-a268-7f130e57d6a3" containerName="maas-api" Sep 08 17:04:36.913393 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:36.913370 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" Sep 08 17:04:36.916581 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:36.916559 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f155bce9-serving-cert\"" Sep 08 17:04:36.926832 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:36.926806 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9"] Sep 08 17:04:36.965321 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:36.965270 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:37.039926 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:37.039895 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69daa404-c01e-47b8-935f-cc63c16ea63e-metrics-certs\") pod \"maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9\" (UID: \"69daa404-c01e-47b8-935f-cc63c16ea63e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" Sep 08 17:04:37.040100 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:37.039946 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jkqj\" (UniqueName: \"kubernetes.io/projected/69daa404-c01e-47b8-935f-cc63c16ea63e-kube-api-access-4jkqj\") pod \"maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9\" (UID: \"69daa404-c01e-47b8-935f-cc63c16ea63e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" Sep 08 17:04:37.040100 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:37.040059 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69daa404-c01e-47b8-935f-cc63c16ea63e-openshift-service-ca\") pod \"maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9\" (UID: \"69daa404-c01e-47b8-935f-cc63c16ea63e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" Sep 08 17:04:37.040100 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:37.040090 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69daa404-c01e-47b8-935f-cc63c16ea63e-maas-api-tls\") pod \"maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9\" (UID: \"69daa404-c01e-47b8-935f-cc63c16ea63e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" Sep 08 17:04:37.066011 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:37.065976 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:37.141093 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:37.141061 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jkqj\" (UniqueName: \"kubernetes.io/projected/69daa404-c01e-47b8-935f-cc63c16ea63e-kube-api-access-4jkqj\") pod \"maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9\" (UID: \"69daa404-c01e-47b8-935f-cc63c16ea63e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" Sep 08 17:04:37.141265 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:37.141122 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69daa404-c01e-47b8-935f-cc63c16ea63e-openshift-service-ca\") pod \"maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9\" (UID: \"69daa404-c01e-47b8-935f-cc63c16ea63e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" Sep 08 17:04:37.141265 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:37.141143 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69daa404-c01e-47b8-935f-cc63c16ea63e-maas-api-tls\") pod \"maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9\" (UID: \"69daa404-c01e-47b8-935f-cc63c16ea63e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" Sep 08 17:04:37.141265 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:37.141174 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69daa404-c01e-47b8-935f-cc63c16ea63e-metrics-certs\") pod \"maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9\" (UID: \"69daa404-c01e-47b8-935f-cc63c16ea63e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" Sep 08 17:04:37.141463 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:04:37.141287 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-serving-cert: secret "maas-api-e2e-ait-f155bce9-serving-cert" not found Sep 08 17:04:37.141463 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:04:37.141348 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69daa404-c01e-47b8-935f-cc63c16ea63e-maas-api-tls podName:69daa404-c01e-47b8-935f-cc63c16ea63e nodeName:}" failed. No retries permitted until 2026-09-08 17:04:37.641329988 +0000 UTC m=+840.378161084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/69daa404-c01e-47b8-935f-cc63c16ea63e-maas-api-tls") pod "maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" (UID: "69daa404-c01e-47b8-935f-cc63c16ea63e") : secret "maas-api-e2e-ait-f155bce9-serving-cert" not found Sep 08 17:04:37.141798 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:37.141779 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69daa404-c01e-47b8-935f-cc63c16ea63e-openshift-service-ca\") pod \"maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9\" (UID: \"69daa404-c01e-47b8-935f-cc63c16ea63e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" Sep 08 17:04:37.143703 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:37.143685 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69daa404-c01e-47b8-935f-cc63c16ea63e-metrics-certs\") pod \"maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9\" (UID: \"69daa404-c01e-47b8-935f-cc63c16ea63e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" Sep 08 17:04:37.150564 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:37.150537 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jkqj\" (UniqueName: \"kubernetes.io/projected/69daa404-c01e-47b8-935f-cc63c16ea63e-kube-api-access-4jkqj\") pod \"maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9\" (UID: \"69daa404-c01e-47b8-935f-cc63c16ea63e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" Sep 08 17:04:37.652740 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:37.652699 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69daa404-c01e-47b8-935f-cc63c16ea63e-maas-api-tls\") pod \"maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9\" (UID: \"69daa404-c01e-47b8-935f-cc63c16ea63e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" Sep 08 17:04:37.655059 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:37.655028 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69daa404-c01e-47b8-935f-cc63c16ea63e-maas-api-tls\") pod \"maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9\" (UID: \"69daa404-c01e-47b8-935f-cc63c16ea63e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" Sep 08 17:04:37.824151 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:37.824111 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" Sep 08 17:04:37.882905 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:37.882559 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:38.094022 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:38.093988 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:38.128449 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:38.128404 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9"] Sep 08 17:04:38.129205 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:04:38.129178 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69daa404_c01e_47b8_935f_cc63c16ea63e.slice/crio-7b2ee5f7a614727559062949d2744ac11c2158c7d48f18335db2b43dab2dc559 WatchSource:0}: Error finding container 7b2ee5f7a614727559062949d2744ac11c2158c7d48f18335db2b43dab2dc559: Status 404 returned error can't find the container with id 7b2ee5f7a614727559062949d2744ac11c2158c7d48f18335db2b43dab2dc559 Sep 08 17:04:38.367138 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:38.367055 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" event={"ID":"69daa404-c01e-47b8-935f-cc63c16ea63e","Type":"ContainerStarted","Data":"7b2ee5f7a614727559062949d2744ac11c2158c7d48f18335db2b43dab2dc559"} Sep 08 17:04:39.375900 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:39.375862 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" event={"ID":"69daa404-c01e-47b8-935f-cc63c16ea63e","Type":"ContainerStarted","Data":"962ccef000ca1abe28605317f90007544c5d635c0400ef37d160e60ff0d6daac"} Sep 08 17:04:39.376292 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:39.375926 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" Sep 08 17:04:39.401358 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:39.401254 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" podStartSLOduration=2.382441589 podStartE2EDuration="3.401234384s" totalImagesPullingTime="1.018792795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:04:39.397996451 +0000 UTC m=+842.134827568" watchObservedRunningTime="2026-09-08 17:04:39.401234384 +0000 UTC m=+842.138065499" Sep 08 17:04:41.174726 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:41.174682 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:45.387878 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:45.387845 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" Sep 08 17:04:57.169467 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:57.169407 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:04:58.544777 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:04:58.544745 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:02.447395 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:02.447355 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:07.626534 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:07.626503 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9"] Sep 08 17:05:07.626934 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:07.626701 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" podUID="69daa404-c01e-47b8-935f-cc63c16ea63e" containerName="maas-api" containerID="cri-o://962ccef000ca1abe28605317f90007544c5d635c0400ef37d160e60ff0d6daac" gracePeriod=30 Sep 08 17:05:07.685866 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:07.685827 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:07.829614 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:07.829571 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:07.893318 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:07.893236 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" Sep 08 17:05:07.941246 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:07.941213 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:08.026282 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:08.026244 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:08.041197 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:08.041166 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69daa404-c01e-47b8-935f-cc63c16ea63e-metrics-certs\") pod \"69daa404-c01e-47b8-935f-cc63c16ea63e\" (UID: \"69daa404-c01e-47b8-935f-cc63c16ea63e\") " Sep 08 17:05:08.041197 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:08.041198 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69daa404-c01e-47b8-935f-cc63c16ea63e-openshift-service-ca\") pod \"69daa404-c01e-47b8-935f-cc63c16ea63e\" (UID: \"69daa404-c01e-47b8-935f-cc63c16ea63e\") " Sep 08 17:05:08.041469 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:08.041215 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69daa404-c01e-47b8-935f-cc63c16ea63e-maas-api-tls\") pod \"69daa404-c01e-47b8-935f-cc63c16ea63e\" (UID: \"69daa404-c01e-47b8-935f-cc63c16ea63e\") " Sep 08 17:05:08.041469 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:08.041306 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jkqj\" (UniqueName: \"kubernetes.io/projected/69daa404-c01e-47b8-935f-cc63c16ea63e-kube-api-access-4jkqj\") pod \"69daa404-c01e-47b8-935f-cc63c16ea63e\" (UID: \"69daa404-c01e-47b8-935f-cc63c16ea63e\") " Sep 08 17:05:08.041636 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:08.041603 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69daa404-c01e-47b8-935f-cc63c16ea63e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "69daa404-c01e-47b8-935f-cc63c16ea63e" (UID: "69daa404-c01e-47b8-935f-cc63c16ea63e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:05:08.043527 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:08.043491 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69daa404-c01e-47b8-935f-cc63c16ea63e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "69daa404-c01e-47b8-935f-cc63c16ea63e" (UID: "69daa404-c01e-47b8-935f-cc63c16ea63e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:05:08.043527 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:08.043509 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69daa404-c01e-47b8-935f-cc63c16ea63e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "69daa404-c01e-47b8-935f-cc63c16ea63e" (UID: "69daa404-c01e-47b8-935f-cc63c16ea63e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:05:08.043677 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:08.043654 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69daa404-c01e-47b8-935f-cc63c16ea63e-kube-api-access-4jkqj" (OuterVolumeSpecName: "kube-api-access-4jkqj") pod "69daa404-c01e-47b8-935f-cc63c16ea63e" (UID: "69daa404-c01e-47b8-935f-cc63c16ea63e"). InnerVolumeSpecName "kube-api-access-4jkqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:05:08.142735 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:08.142703 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jkqj\" (UniqueName: \"kubernetes.io/projected/69daa404-c01e-47b8-935f-cc63c16ea63e-kube-api-access-4jkqj\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:05:08.142735 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:08.142733 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69daa404-c01e-47b8-935f-cc63c16ea63e-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:05:08.142918 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:08.142745 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69daa404-c01e-47b8-935f-cc63c16ea63e-openshift-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:05:08.142918 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:08.142753 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69daa404-c01e-47b8-935f-cc63c16ea63e-maas-api-tls\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:05:08.554771 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:08.554736 2565 generic.go:356] "Generic (PLEG): container finished" podID="69daa404-c01e-47b8-935f-cc63c16ea63e" containerID="962ccef000ca1abe28605317f90007544c5d635c0400ef37d160e60ff0d6daac" exitCode=0 Sep 08 17:05:08.554937 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:08.554837 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" Sep 08 17:05:08.554937 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:08.554875 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" event={"ID":"69daa404-c01e-47b8-935f-cc63c16ea63e","Type":"ContainerDied","Data":"962ccef000ca1abe28605317f90007544c5d635c0400ef37d160e60ff0d6daac"} Sep 08 17:05:08.554937 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:08.554923 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9" event={"ID":"69daa404-c01e-47b8-935f-cc63c16ea63e","Type":"ContainerDied","Data":"7b2ee5f7a614727559062949d2744ac11c2158c7d48f18335db2b43dab2dc559"} Sep 08 17:05:08.555070 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:08.554948 2565 scope.go:117] "RemoveContainer" containerID="962ccef000ca1abe28605317f90007544c5d635c0400ef37d160e60ff0d6daac" Sep 08 17:05:08.565574 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:08.565557 2565 scope.go:117] "RemoveContainer" containerID="962ccef000ca1abe28605317f90007544c5d635c0400ef37d160e60ff0d6daac" Sep 08 17:05:08.565828 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:05:08.565808 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"962ccef000ca1abe28605317f90007544c5d635c0400ef37d160e60ff0d6daac\": container with ID starting with 962ccef000ca1abe28605317f90007544c5d635c0400ef37d160e60ff0d6daac not found: ID does not exist" containerID="962ccef000ca1abe28605317f90007544c5d635c0400ef37d160e60ff0d6daac" Sep 08 17:05:08.565894 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:08.565834 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962ccef000ca1abe28605317f90007544c5d635c0400ef37d160e60ff0d6daac"} err="failed to get container status \"962ccef000ca1abe28605317f90007544c5d635c0400ef37d160e60ff0d6daac\": rpc error: code = NotFound desc = could not find container \"962ccef000ca1abe28605317f90007544c5d635c0400ef37d160e60ff0d6daac\": container with ID starting with 962ccef000ca1abe28605317f90007544c5d635c0400ef37d160e60ff0d6daac not found: ID does not exist" Sep 08 17:05:08.581950 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:08.581929 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9"] Sep 08 17:05:08.585704 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:08.585682 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f155bce9-54c4b9cd44-t7qt9"] Sep 08 17:05:09.139276 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:09.139243 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:09.593447 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:09.593381 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:09.760595 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:09.760563 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69daa404-c01e-47b8-935f-cc63c16ea63e" path="/var/lib/kubelet/pods/69daa404-c01e-47b8-935f-cc63c16ea63e/volumes" Sep 08 17:05:09.929344 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:09.929310 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:10.190557 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:10.190459 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:11.967620 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:11.967586 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:12.049573 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:12.049536 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:13.355764 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:13.355723 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk"] Sep 08 17:05:13.356302 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:13.356282 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69daa404-c01e-47b8-935f-cc63c16ea63e" containerName="maas-api" Sep 08 17:05:13.356353 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:13.356304 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="69daa404-c01e-47b8-935f-cc63c16ea63e" containerName="maas-api" Sep 08 17:05:13.356389 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:13.356385 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="69daa404-c01e-47b8-935f-cc63c16ea63e" containerName="maas-api" Sep 08 17:05:13.360847 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:13.360830 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" Sep 08 17:05:13.363640 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:13.363619 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-b709328f-serving-cert\"" Sep 08 17:05:13.375307 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:13.375287 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk"] Sep 08 17:05:13.401919 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:13.401882 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:13.493142 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:13.493110 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29bdd3f4-9192-4e65-94ab-d3782accfd60-maas-api-tls\") pod \"maas-api-e2e-derive-b709328f-77dd887dbd-k82tk\" (UID: \"29bdd3f4-9192-4e65-94ab-d3782accfd60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" Sep 08 17:05:13.493300 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:13.493156 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29bdd3f4-9192-4e65-94ab-d3782accfd60-openshift-service-ca\") pod \"maas-api-e2e-derive-b709328f-77dd887dbd-k82tk\" (UID: \"29bdd3f4-9192-4e65-94ab-d3782accfd60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" Sep 08 17:05:13.493300 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:13.493176 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29bdd3f4-9192-4e65-94ab-d3782accfd60-metrics-certs\") pod \"maas-api-e2e-derive-b709328f-77dd887dbd-k82tk\" (UID: \"29bdd3f4-9192-4e65-94ab-d3782accfd60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" Sep 08 17:05:13.493397 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:13.493314 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpcs5\" (UniqueName: \"kubernetes.io/projected/29bdd3f4-9192-4e65-94ab-d3782accfd60-kube-api-access-hpcs5\") pod \"maas-api-e2e-derive-b709328f-77dd887dbd-k82tk\" (UID: \"29bdd3f4-9192-4e65-94ab-d3782accfd60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" Sep 08 17:05:13.497500 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:13.497475 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:13.594454 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:13.594403 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpcs5\" (UniqueName: \"kubernetes.io/projected/29bdd3f4-9192-4e65-94ab-d3782accfd60-kube-api-access-hpcs5\") pod \"maas-api-e2e-derive-b709328f-77dd887dbd-k82tk\" (UID: \"29bdd3f4-9192-4e65-94ab-d3782accfd60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" Sep 08 17:05:13.594606 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:13.594481 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29bdd3f4-9192-4e65-94ab-d3782accfd60-maas-api-tls\") pod \"maas-api-e2e-derive-b709328f-77dd887dbd-k82tk\" (UID: \"29bdd3f4-9192-4e65-94ab-d3782accfd60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" Sep 08 17:05:13.594606 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:13.594524 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29bdd3f4-9192-4e65-94ab-d3782accfd60-openshift-service-ca\") pod \"maas-api-e2e-derive-b709328f-77dd887dbd-k82tk\" (UID: \"29bdd3f4-9192-4e65-94ab-d3782accfd60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" Sep 08 17:05:13.594606 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:13.594547 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29bdd3f4-9192-4e65-94ab-d3782accfd60-metrics-certs\") pod \"maas-api-e2e-derive-b709328f-77dd887dbd-k82tk\" (UID: \"29bdd3f4-9192-4e65-94ab-d3782accfd60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" Sep 08 17:05:13.594737 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:05:13.594634 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-serving-cert: secret "maas-api-e2e-derive-b709328f-serving-cert" not found Sep 08 17:05:13.594737 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:05:13.594718 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29bdd3f4-9192-4e65-94ab-d3782accfd60-maas-api-tls podName:29bdd3f4-9192-4e65-94ab-d3782accfd60 nodeName:}" failed. No retries permitted until 2026-09-08 17:05:14.094697801 +0000 UTC m=+876.831528900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/29bdd3f4-9192-4e65-94ab-d3782accfd60-maas-api-tls") pod "maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" (UID: "29bdd3f4-9192-4e65-94ab-d3782accfd60") : secret "maas-api-e2e-derive-b709328f-serving-cert" not found Sep 08 17:05:13.595225 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:13.595210 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29bdd3f4-9192-4e65-94ab-d3782accfd60-openshift-service-ca\") pod \"maas-api-e2e-derive-b709328f-77dd887dbd-k82tk\" (UID: \"29bdd3f4-9192-4e65-94ab-d3782accfd60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" Sep 08 17:05:13.597041 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:13.597024 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29bdd3f4-9192-4e65-94ab-d3782accfd60-metrics-certs\") pod \"maas-api-e2e-derive-b709328f-77dd887dbd-k82tk\" (UID: \"29bdd3f4-9192-4e65-94ab-d3782accfd60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" Sep 08 17:05:13.605358 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:13.605337 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpcs5\" (UniqueName: \"kubernetes.io/projected/29bdd3f4-9192-4e65-94ab-d3782accfd60-kube-api-access-hpcs5\") pod \"maas-api-e2e-derive-b709328f-77dd887dbd-k82tk\" (UID: \"29bdd3f4-9192-4e65-94ab-d3782accfd60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" Sep 08 17:05:13.937075 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:13.937044 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:14.033944 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:14.033915 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:14.100388 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:14.100354 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29bdd3f4-9192-4e65-94ab-d3782accfd60-maas-api-tls\") pod \"maas-api-e2e-derive-b709328f-77dd887dbd-k82tk\" (UID: \"29bdd3f4-9192-4e65-94ab-d3782accfd60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" Sep 08 17:05:14.100570 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:05:14.100506 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-serving-cert: secret "maas-api-e2e-derive-b709328f-serving-cert" not found Sep 08 17:05:14.100615 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:05:14.100591 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29bdd3f4-9192-4e65-94ab-d3782accfd60-maas-api-tls podName:29bdd3f4-9192-4e65-94ab-d3782accfd60 nodeName:}" failed. No retries permitted until 2026-09-08 17:05:15.100576813 +0000 UTC m=+877.837407907 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/29bdd3f4-9192-4e65-94ab-d3782accfd60-maas-api-tls") pod "maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" (UID: "29bdd3f4-9192-4e65-94ab-d3782accfd60") : secret "maas-api-e2e-derive-b709328f-serving-cert" not found Sep 08 17:05:14.460110 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:14.460049 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:15.110215 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:15.110166 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29bdd3f4-9192-4e65-94ab-d3782accfd60-maas-api-tls\") pod \"maas-api-e2e-derive-b709328f-77dd887dbd-k82tk\" (UID: \"29bdd3f4-9192-4e65-94ab-d3782accfd60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" Sep 08 17:05:15.112469 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:15.112447 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29bdd3f4-9192-4e65-94ab-d3782accfd60-maas-api-tls\") pod \"maas-api-e2e-derive-b709328f-77dd887dbd-k82tk\" (UID: \"29bdd3f4-9192-4e65-94ab-d3782accfd60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" Sep 08 17:05:15.172842 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:15.172813 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" Sep 08 17:05:15.447379 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:15.447353 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk"] Sep 08 17:05:15.449979 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:05:15.449935 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29bdd3f4_9192_4e65_94ab_d3782accfd60.slice/crio-cc3e5e868a434cc611eb930fc0e9efdec7d6ec9b72bbfa94a5afa264809d1c36 WatchSource:0}: Error finding container cc3e5e868a434cc611eb930fc0e9efdec7d6ec9b72bbfa94a5afa264809d1c36: Status 404 returned error can't find the container with id cc3e5e868a434cc611eb930fc0e9efdec7d6ec9b72bbfa94a5afa264809d1c36 Sep 08 17:05:15.601112 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:15.601084 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" event={"ID":"29bdd3f4-9192-4e65-94ab-d3782accfd60","Type":"ContainerStarted","Data":"cc3e5e868a434cc611eb930fc0e9efdec7d6ec9b72bbfa94a5afa264809d1c36"} Sep 08 17:05:17.614802 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:17.614755 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" event={"ID":"29bdd3f4-9192-4e65-94ab-d3782accfd60","Type":"ContainerStarted","Data":"63fbe9546d12f927a924bcfebcff1dfc859a2042bf43663de76249d8866aae8a"} Sep 08 17:05:17.615176 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:17.614835 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" Sep 08 17:05:17.632958 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:17.632911 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" podStartSLOduration=3.4798328720000002 podStartE2EDuration="4.632896911s" totalImagesPullingTime="1.153064039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:05:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:05:17.630969719 +0000 UTC m=+880.367800836" watchObservedRunningTime="2026-09-08 17:05:17.632896911 +0000 UTC m=+880.369728026" Sep 08 17:05:23.625617 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:23.625586 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" Sep 08 17:05:32.196941 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:32.196841 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:34.318979 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:34.318944 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:35.503975 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:35.503932 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:36.579043 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:36.579008 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:36.675334 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:36.675289 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:37.184574 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:37.184542 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:37.783515 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:37.783483 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/2.log" Sep 08 17:05:37.784156 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:37.784129 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/2.log" Sep 08 17:05:37.792332 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:37.792309 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xxnl_bf53ac63-f864-402b-bf12-45b289e7e13d/ovn-acl-logging/0.log" Sep 08 17:05:37.792481 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:37.792360 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xxnl_bf53ac63-f864-402b-bf12-45b289e7e13d/ovn-acl-logging/0.log" Sep 08 17:05:38.130738 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:38.130649 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:43.772855 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:43.772818 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:44.403540 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.403498 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk"] Sep 08 17:05:44.403836 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.403721 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" podUID="29bdd3f4-9192-4e65-94ab-d3782accfd60" containerName="maas-api" containerID="cri-o://63fbe9546d12f927a924bcfebcff1dfc859a2042bf43663de76249d8866aae8a" gracePeriod=30 Sep 08 17:05:44.435053 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.435026 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:44.518314 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.518281 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:44.583917 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.583870 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:44.657721 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.657652 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" Sep 08 17:05:44.658000 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.657976 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:44.725302 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.725275 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:44.762407 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.762377 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29bdd3f4-9192-4e65-94ab-d3782accfd60-openshift-service-ca\") pod \"29bdd3f4-9192-4e65-94ab-d3782accfd60\" (UID: \"29bdd3f4-9192-4e65-94ab-d3782accfd60\") " Sep 08 17:05:44.762552 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.762456 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29bdd3f4-9192-4e65-94ab-d3782accfd60-metrics-certs\") pod \"29bdd3f4-9192-4e65-94ab-d3782accfd60\" (UID: \"29bdd3f4-9192-4e65-94ab-d3782accfd60\") " Sep 08 17:05:44.762597 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.762553 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpcs5\" (UniqueName: \"kubernetes.io/projected/29bdd3f4-9192-4e65-94ab-d3782accfd60-kube-api-access-hpcs5\") pod \"29bdd3f4-9192-4e65-94ab-d3782accfd60\" (UID: \"29bdd3f4-9192-4e65-94ab-d3782accfd60\") " Sep 08 17:05:44.762644 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.762618 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29bdd3f4-9192-4e65-94ab-d3782accfd60-maas-api-tls\") pod \"29bdd3f4-9192-4e65-94ab-d3782accfd60\" (UID: \"29bdd3f4-9192-4e65-94ab-d3782accfd60\") " Sep 08 17:05:44.762767 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.762745 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29bdd3f4-9192-4e65-94ab-d3782accfd60-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "29bdd3f4-9192-4e65-94ab-d3782accfd60" (UID: "29bdd3f4-9192-4e65-94ab-d3782accfd60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:05:44.762974 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.762956 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29bdd3f4-9192-4e65-94ab-d3782accfd60-openshift-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:05:44.764747 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.764719 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29bdd3f4-9192-4e65-94ab-d3782accfd60-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "29bdd3f4-9192-4e65-94ab-d3782accfd60" (UID: "29bdd3f4-9192-4e65-94ab-d3782accfd60"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:05:44.764843 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.764786 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29bdd3f4-9192-4e65-94ab-d3782accfd60-kube-api-access-hpcs5" (OuterVolumeSpecName: "kube-api-access-hpcs5") pod "29bdd3f4-9192-4e65-94ab-d3782accfd60" (UID: "29bdd3f4-9192-4e65-94ab-d3782accfd60"). InnerVolumeSpecName "kube-api-access-hpcs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:05:44.764843 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.764794 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29bdd3f4-9192-4e65-94ab-d3782accfd60-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "29bdd3f4-9192-4e65-94ab-d3782accfd60" (UID: "29bdd3f4-9192-4e65-94ab-d3782accfd60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:05:44.783262 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.783231 2565 generic.go:356] "Generic (PLEG): container finished" podID="29bdd3f4-9192-4e65-94ab-d3782accfd60" containerID="63fbe9546d12f927a924bcfebcff1dfc859a2042bf43663de76249d8866aae8a" exitCode=0 Sep 08 17:05:44.783661 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.783285 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" Sep 08 17:05:44.783661 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.783286 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" event={"ID":"29bdd3f4-9192-4e65-94ab-d3782accfd60","Type":"ContainerDied","Data":"63fbe9546d12f927a924bcfebcff1dfc859a2042bf43663de76249d8866aae8a"} Sep 08 17:05:44.783661 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.783351 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk" event={"ID":"29bdd3f4-9192-4e65-94ab-d3782accfd60","Type":"ContainerDied","Data":"cc3e5e868a434cc611eb930fc0e9efdec7d6ec9b72bbfa94a5afa264809d1c36"} Sep 08 17:05:44.783661 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.783382 2565 scope.go:117] "RemoveContainer" containerID="63fbe9546d12f927a924bcfebcff1dfc859a2042bf43663de76249d8866aae8a" Sep 08 17:05:44.796008 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.795901 2565 scope.go:117] "RemoveContainer" containerID="63fbe9546d12f927a924bcfebcff1dfc859a2042bf43663de76249d8866aae8a" Sep 08 17:05:44.796362 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:05:44.796334 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63fbe9546d12f927a924bcfebcff1dfc859a2042bf43663de76249d8866aae8a\": container with ID starting with 63fbe9546d12f927a924bcfebcff1dfc859a2042bf43663de76249d8866aae8a not found: ID does not exist" containerID="63fbe9546d12f927a924bcfebcff1dfc859a2042bf43663de76249d8866aae8a" Sep 08 17:05:44.796492 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.796372 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63fbe9546d12f927a924bcfebcff1dfc859a2042bf43663de76249d8866aae8a"} err="failed to get container status \"63fbe9546d12f927a924bcfebcff1dfc859a2042bf43663de76249d8866aae8a\": rpc error: code = NotFound desc = could not find container \"63fbe9546d12f927a924bcfebcff1dfc859a2042bf43663de76249d8866aae8a\": container with ID starting with 63fbe9546d12f927a924bcfebcff1dfc859a2042bf43663de76249d8866aae8a not found: ID does not exist" Sep 08 17:05:44.810356 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.810333 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk"] Sep 08 17:05:44.820860 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.820831 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b709328f-77dd887dbd-k82tk"] Sep 08 17:05:44.826512 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.824135 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:44.864272 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.864250 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29bdd3f4-9192-4e65-94ab-d3782accfd60-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:05:44.864272 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.864273 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpcs5\" (UniqueName: \"kubernetes.io/projected/29bdd3f4-9192-4e65-94ab-d3782accfd60-kube-api-access-hpcs5\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:05:44.864402 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:44.864282 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29bdd3f4-9192-4e65-94ab-d3782accfd60-maas-api-tls\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:05:45.516973 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:45.516940 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:45.609700 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:45.609668 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:45.759932 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:45.759899 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29bdd3f4-9192-4e65-94ab-d3782accfd60" path="/var/lib/kubelet/pods/29bdd3f4-9192-4e65-94ab-d3782accfd60/volumes" Sep 08 17:05:47.102459 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.102403 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx"] Sep 08 17:05:47.102883 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.102685 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" podUID="a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc" containerName="maas-api" containerID="cri-o://a5326241c404f717ba354166fc172dfd9b59de39db19576d9680ef5465cf8045" gracePeriod=30 Sep 08 17:05:47.364874 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.364814 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" Sep 08 17:05:47.430419 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.430387 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:47.488071 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.488046 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-openshift-service-ca\") pod \"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc\" (UID: \"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc\") " Sep 08 17:05:47.488215 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.488101 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-maas-api-tls\") pod \"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc\" (UID: \"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc\") " Sep 08 17:05:47.488215 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.488184 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-metrics-certs\") pod \"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc\" (UID: \"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc\") " Sep 08 17:05:47.488355 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.488218 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smg66\" (UniqueName: \"kubernetes.io/projected/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-kube-api-access-smg66\") pod \"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc\" (UID: \"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc\") " Sep 08 17:05:47.488396 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.488361 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc" (UID: "a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:05:47.488532 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.488509 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-openshift-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:05:47.490242 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.490213 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc" (UID: "a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:05:47.490326 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.490262 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc" (UID: "a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:05:47.490326 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.490262 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-kube-api-access-smg66" (OuterVolumeSpecName: "kube-api-access-smg66") pod "a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc" (UID: "a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc"). InnerVolumeSpecName "kube-api-access-smg66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:05:47.522570 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.522500 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:47.589714 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.589692 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-maas-api-tls\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:05:47.589714 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.589713 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:05:47.589855 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.589722 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smg66\" (UniqueName: \"kubernetes.io/projected/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc-kube-api-access-smg66\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:05:47.802073 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.802048 2565 generic.go:356] "Generic (PLEG): container finished" podID="a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc" containerID="a5326241c404f717ba354166fc172dfd9b59de39db19576d9680ef5465cf8045" exitCode=0 Sep 08 17:05:47.802189 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.802086 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" Sep 08 17:05:47.802189 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.802131 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" event={"ID":"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc","Type":"ContainerDied","Data":"a5326241c404f717ba354166fc172dfd9b59de39db19576d9680ef5465cf8045"} Sep 08 17:05:47.802189 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.802184 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx" event={"ID":"a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc","Type":"ContainerDied","Data":"b9c8fffc68a03664624c5656c74fd2d0fca8961f23fc6a83d4326669a6c0f4b4"} Sep 08 17:05:47.802362 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.802216 2565 scope.go:117] "RemoveContainer" containerID="a5326241c404f717ba354166fc172dfd9b59de39db19576d9680ef5465cf8045" Sep 08 17:05:47.811260 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.811243 2565 scope.go:117] "RemoveContainer" containerID="a5326241c404f717ba354166fc172dfd9b59de39db19576d9680ef5465cf8045" Sep 08 17:05:47.811537 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:05:47.811516 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5326241c404f717ba354166fc172dfd9b59de39db19576d9680ef5465cf8045\": container with ID starting with a5326241c404f717ba354166fc172dfd9b59de39db19576d9680ef5465cf8045 not found: ID does not exist" containerID="a5326241c404f717ba354166fc172dfd9b59de39db19576d9680ef5465cf8045" Sep 08 17:05:47.811602 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.811550 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5326241c404f717ba354166fc172dfd9b59de39db19576d9680ef5465cf8045"} err="failed to get container status \"a5326241c404f717ba354166fc172dfd9b59de39db19576d9680ef5465cf8045\": rpc error: code = NotFound desc = could not find container \"a5326241c404f717ba354166fc172dfd9b59de39db19576d9680ef5465cf8045\": container with ID starting with a5326241c404f717ba354166fc172dfd9b59de39db19576d9680ef5465cf8045 not found: ID does not exist" Sep 08 17:05:47.843815 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.843790 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx"] Sep 08 17:05:47.849847 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.849819 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:47.850394 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:47.850374 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-66e7a8-67d5575d48-89tmx"] Sep 08 17:05:49.760599 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:49.760563 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc" path="/var/lib/kubelet/pods/a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc/volumes" Sep 08 17:05:53.495731 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:53.495694 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:53.603009 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:53.602975 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:53.710083 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:53.710050 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:53.813411 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:53.813328 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:54.195831 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:54.195783 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:54.744501 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:54.744472 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:54.836227 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:54.836189 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:55.015148 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:55.015057 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:55.454509 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:55.454473 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:55.786078 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:55.785995 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:56.110048 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:56.109964 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:58.822364 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:58.822314 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:59.706903 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:59.706865 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:05:59.934516 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:59.934482 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s"] Sep 08 17:05:59.934884 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:59.934872 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29bdd3f4-9192-4e65-94ab-d3782accfd60" containerName="maas-api" Sep 08 17:05:59.934928 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:59.934885 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="29bdd3f4-9192-4e65-94ab-d3782accfd60" containerName="maas-api" Sep 08 17:05:59.934928 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:59.934915 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc" containerName="maas-api" Sep 08 17:05:59.934928 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:59.934923 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc" containerName="maas-api" Sep 08 17:05:59.935021 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:59.934980 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a83d7a8b-b0b2-44e6-87b4-dc8d105ab7cc" containerName="maas-api" Sep 08 17:05:59.935021 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:59.934990 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="29bdd3f4-9192-4e65-94ab-d3782accfd60" containerName="maas-api" Sep 08 17:05:59.940150 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:59.940125 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" Sep 08 17:05:59.942514 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:59.942489 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-3b5d3c-serving-cert\"" Sep 08 17:05:59.953616 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:05:59.953589 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s"] Sep 08 17:06:00.000897 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:00.000822 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:06:00.113402 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:00.113362 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbvqc\" (UniqueName: \"kubernetes.io/projected/9046b27a-6da6-4878-8487-f845f6eed562-kube-api-access-xbvqc\") pod \"maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s\" (UID: \"9046b27a-6da6-4878-8487-f845f6eed562\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" Sep 08 17:06:00.113625 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:00.113441 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9046b27a-6da6-4878-8487-f845f6eed562-maas-api-tls\") pod \"maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s\" (UID: \"9046b27a-6da6-4878-8487-f845f6eed562\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" Sep 08 17:06:00.113625 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:00.113468 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9046b27a-6da6-4878-8487-f845f6eed562-openshift-service-ca\") pod \"maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s\" (UID: \"9046b27a-6da6-4878-8487-f845f6eed562\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" Sep 08 17:06:00.113625 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:00.113542 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9046b27a-6da6-4878-8487-f845f6eed562-metrics-certs\") pod \"maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s\" (UID: \"9046b27a-6da6-4878-8487-f845f6eed562\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" Sep 08 17:06:00.214612 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:00.214574 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9046b27a-6da6-4878-8487-f845f6eed562-maas-api-tls\") pod \"maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s\" (UID: \"9046b27a-6da6-4878-8487-f845f6eed562\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" Sep 08 17:06:00.214612 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:00.214616 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9046b27a-6da6-4878-8487-f845f6eed562-openshift-service-ca\") pod \"maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s\" (UID: \"9046b27a-6da6-4878-8487-f845f6eed562\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" Sep 08 17:06:00.214852 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:00.214652 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9046b27a-6da6-4878-8487-f845f6eed562-metrics-certs\") pod \"maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s\" (UID: \"9046b27a-6da6-4878-8487-f845f6eed562\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" Sep 08 17:06:00.214852 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:00.214698 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbvqc\" (UniqueName: \"kubernetes.io/projected/9046b27a-6da6-4878-8487-f845f6eed562-kube-api-access-xbvqc\") pod \"maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s\" (UID: \"9046b27a-6da6-4878-8487-f845f6eed562\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" Sep 08 17:06:00.215388 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:00.215362 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9046b27a-6da6-4878-8487-f845f6eed562-openshift-service-ca\") pod \"maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s\" (UID: \"9046b27a-6da6-4878-8487-f845f6eed562\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" Sep 08 17:06:00.216923 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:00.216894 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9046b27a-6da6-4878-8487-f845f6eed562-maas-api-tls\") pod \"maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s\" (UID: \"9046b27a-6da6-4878-8487-f845f6eed562\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" Sep 08 17:06:00.217067 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:00.217050 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9046b27a-6da6-4878-8487-f845f6eed562-metrics-certs\") pod \"maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s\" (UID: \"9046b27a-6da6-4878-8487-f845f6eed562\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" Sep 08 17:06:00.224612 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:00.224594 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbvqc\" (UniqueName: \"kubernetes.io/projected/9046b27a-6da6-4878-8487-f845f6eed562-kube-api-access-xbvqc\") pod \"maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s\" (UID: \"9046b27a-6da6-4878-8487-f845f6eed562\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" Sep 08 17:06:00.255255 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:00.255198 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" Sep 08 17:06:00.542638 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:00.542608 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s"] Sep 08 17:06:00.544544 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:06:00.544516 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9046b27a_6da6_4878_8487_f845f6eed562.slice/crio-e5db9217e53162c2b33d2f91b9a6dc43f86e64fbd36513ca7d74dad6fed842a6 WatchSource:0}: Error finding container e5db9217e53162c2b33d2f91b9a6dc43f86e64fbd36513ca7d74dad6fed842a6: Status 404 returned error can't find the container with id e5db9217e53162c2b33d2f91b9a6dc43f86e64fbd36513ca7d74dad6fed842a6 Sep 08 17:06:00.877645 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:00.877561 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" event={"ID":"9046b27a-6da6-4878-8487-f845f6eed562","Type":"ContainerStarted","Data":"e5db9217e53162c2b33d2f91b9a6dc43f86e64fbd36513ca7d74dad6fed842a6"} Sep 08 17:06:00.889890 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:00.889861 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:06:01.050685 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:01.050652 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:06:01.885112 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:01.885036 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" event={"ID":"9046b27a-6da6-4878-8487-f845f6eed562","Type":"ContainerStarted","Data":"dbf9e74fdc583f8c8255f2a290879dedf20b13b250549284cf4082d101ac0c26"} Sep 08 17:06:01.885291 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:01.885152 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" Sep 08 17:06:07.896015 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:07.895983 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" Sep 08 17:06:07.912842 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:07.912795 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" podStartSLOduration=7.957028248 podStartE2EDuration="8.912775708s" totalImagesPullingTime="955.74746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:05:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:06:01.919387805 +0000 UTC m=+924.656218921" watchObservedRunningTime="2026-09-08 17:06:07.912775708 +0000 UTC m=+930.649606823" Sep 08 17:06:11.844140 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:11.844099 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:06:20.145054 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:20.145018 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:06:23.743592 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:23.743548 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:06:26.088114 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:26.088080 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:06:26.198751 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:26.198719 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:06:26.316682 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:26.316638 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:06:36.151828 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:36.151783 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:06:42.118802 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:42.118767 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:06:47.480975 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:47.480936 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:06:48.586668 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:48.586636 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:06:48.699373 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:48.699327 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:06:48.788342 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:48.788311 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:06:58.164726 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:58.164692 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s"] Sep 08 17:06:58.165197 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:58.164944 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" podUID="9046b27a-6da6-4878-8487-f845f6eed562" containerName="maas-api" containerID="cri-o://dbf9e74fdc583f8c8255f2a290879dedf20b13b250549284cf4082d101ac0c26" gracePeriod=30 Sep 08 17:06:58.415443 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:58.415346 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:06:58.421700 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:58.421676 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" Sep 08 17:06:58.505290 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:58.505264 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:06:58.529737 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:58.529710 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbvqc\" (UniqueName: \"kubernetes.io/projected/9046b27a-6da6-4878-8487-f845f6eed562-kube-api-access-xbvqc\") pod \"9046b27a-6da6-4878-8487-f845f6eed562\" (UID: \"9046b27a-6da6-4878-8487-f845f6eed562\") " Sep 08 17:06:58.529869 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:58.529811 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9046b27a-6da6-4878-8487-f845f6eed562-maas-api-tls\") pod \"9046b27a-6da6-4878-8487-f845f6eed562\" (UID: \"9046b27a-6da6-4878-8487-f845f6eed562\") " Sep 08 17:06:58.529934 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:58.529874 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9046b27a-6da6-4878-8487-f845f6eed562-openshift-service-ca\") pod \"9046b27a-6da6-4878-8487-f845f6eed562\" (UID: \"9046b27a-6da6-4878-8487-f845f6eed562\") " Sep 08 17:06:58.529934 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:58.529910 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9046b27a-6da6-4878-8487-f845f6eed562-metrics-certs\") pod \"9046b27a-6da6-4878-8487-f845f6eed562\" (UID: \"9046b27a-6da6-4878-8487-f845f6eed562\") " Sep 08 17:06:58.530183 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:58.530156 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9046b27a-6da6-4878-8487-f845f6eed562-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9046b27a-6da6-4878-8487-f845f6eed562" (UID: "9046b27a-6da6-4878-8487-f845f6eed562"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:06:58.531927 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:58.531899 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9046b27a-6da6-4878-8487-f845f6eed562-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9046b27a-6da6-4878-8487-f845f6eed562" (UID: "9046b27a-6da6-4878-8487-f845f6eed562"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:06:58.532015 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:58.531922 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9046b27a-6da6-4878-8487-f845f6eed562-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9046b27a-6da6-4878-8487-f845f6eed562" (UID: "9046b27a-6da6-4878-8487-f845f6eed562"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:06:58.532015 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:58.531930 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9046b27a-6da6-4878-8487-f845f6eed562-kube-api-access-xbvqc" (OuterVolumeSpecName: "kube-api-access-xbvqc") pod "9046b27a-6da6-4878-8487-f845f6eed562" (UID: "9046b27a-6da6-4878-8487-f845f6eed562"). InnerVolumeSpecName "kube-api-access-xbvqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:06:58.631562 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:58.631537 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbvqc\" (UniqueName: \"kubernetes.io/projected/9046b27a-6da6-4878-8487-f845f6eed562-kube-api-access-xbvqc\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:06:58.631562 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:58.631558 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9046b27a-6da6-4878-8487-f845f6eed562-maas-api-tls\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:06:58.631700 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:58.631568 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9046b27a-6da6-4878-8487-f845f6eed562-openshift-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:06:58.631700 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:58.631578 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9046b27a-6da6-4878-8487-f845f6eed562-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:06:58.832259 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:58.832226 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:06:59.245485 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:59.245419 2565 generic.go:356] "Generic (PLEG): container finished" podID="9046b27a-6da6-4878-8487-f845f6eed562" containerID="dbf9e74fdc583f8c8255f2a290879dedf20b13b250549284cf4082d101ac0c26" exitCode=0 Sep 08 17:06:59.245485 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:59.245465 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" event={"ID":"9046b27a-6da6-4878-8487-f845f6eed562","Type":"ContainerDied","Data":"dbf9e74fdc583f8c8255f2a290879dedf20b13b250549284cf4082d101ac0c26"} Sep 08 17:06:59.246048 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:59.245498 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" Sep 08 17:06:59.246048 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:59.245537 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s" event={"ID":"9046b27a-6da6-4878-8487-f845f6eed562","Type":"ContainerDied","Data":"e5db9217e53162c2b33d2f91b9a6dc43f86e64fbd36513ca7d74dad6fed842a6"} Sep 08 17:06:59.246048 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:59.245572 2565 scope.go:117] "RemoveContainer" containerID="dbf9e74fdc583f8c8255f2a290879dedf20b13b250549284cf4082d101ac0c26" Sep 08 17:06:59.255103 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:59.255074 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:06:59.256689 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:59.256671 2565 scope.go:117] "RemoveContainer" containerID="dbf9e74fdc583f8c8255f2a290879dedf20b13b250549284cf4082d101ac0c26" Sep 08 17:06:59.257000 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:06:59.256967 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf9e74fdc583f8c8255f2a290879dedf20b13b250549284cf4082d101ac0c26\": container with ID starting with dbf9e74fdc583f8c8255f2a290879dedf20b13b250549284cf4082d101ac0c26 not found: ID does not exist" containerID="dbf9e74fdc583f8c8255f2a290879dedf20b13b250549284cf4082d101ac0c26" Sep 08 17:06:59.257097 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:59.257008 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf9e74fdc583f8c8255f2a290879dedf20b13b250549284cf4082d101ac0c26"} err="failed to get container status \"dbf9e74fdc583f8c8255f2a290879dedf20b13b250549284cf4082d101ac0c26\": rpc error: code = NotFound desc = could not find container \"dbf9e74fdc583f8c8255f2a290879dedf20b13b250549284cf4082d101ac0c26\": container with ID starting with dbf9e74fdc583f8c8255f2a290879dedf20b13b250549284cf4082d101ac0c26 not found: ID does not exist" Sep 08 17:06:59.271569 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:59.271545 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s"] Sep 08 17:06:59.274078 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:59.274059 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3b5d3c-654fdc8f8d-ljc5s"] Sep 08 17:06:59.762940 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:06:59.762905 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9046b27a-6da6-4878-8487-f845f6eed562" path="/var/lib/kubelet/pods/9046b27a-6da6-4878-8487-f845f6eed562/volumes" Sep 08 17:07:00.140440 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:00.140295 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:04.205437 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:04.205385 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:06.073656 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:06.073621 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:06.140878 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:06.140848 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:06.225953 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:06.225897 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:06.326859 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:06.326777 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:07.019586 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:07.019539 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:07.108506 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:07.108476 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:07.208130 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:07.208101 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:08.695585 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:08.695552 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:09.739785 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:09.739746 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:10.021131 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:10.021044 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:10.458199 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:10.458160 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:10.897418 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:10.897337 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:11.192830 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:11.192799 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:12.561187 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:12.561147 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:12.663676 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:12.663650 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:12.767358 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:12.767329 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:14.725439 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:14.725397 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:14.826835 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:14.826802 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:15.413175 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:15.413141 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:18.742184 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:18.742153 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:20.849070 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:20.849015 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:21.710700 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:21.710658 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:21.811283 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:21.811230 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:21.940820 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:21.940786 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:22.311931 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:22.311898 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:22.815890 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:22.815854 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:25.277978 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:25.277944 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:25.362533 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:25.362497 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:25.459860 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:25.459830 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:26.533912 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:26.533870 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff"] Sep 08 17:07:26.534282 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:26.534274 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9046b27a-6da6-4878-8487-f845f6eed562" containerName="maas-api" Sep 08 17:07:26.534324 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:26.534285 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9046b27a-6da6-4878-8487-f845f6eed562" containerName="maas-api" Sep 08 17:07:26.534370 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:26.534359 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9046b27a-6da6-4878-8487-f845f6eed562" containerName="maas-api" Sep 08 17:07:26.537652 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:26.537634 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" Sep 08 17:07:26.540912 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:26.540883 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a837a8a9-serving-cert\"" Sep 08 17:07:26.549624 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:26.549593 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff"] Sep 08 17:07:26.576347 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:26.576315 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-metrics-certs\") pod \"maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff\" (UID: \"1950d218-889e-49d8-8bb8-11b2fc4d6ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" Sep 08 17:07:26.576540 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:26.576372 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-maas-api-tls\") pod \"maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff\" (UID: \"1950d218-889e-49d8-8bb8-11b2fc4d6ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" Sep 08 17:07:26.576540 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:26.576491 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg7z6\" (UniqueName: \"kubernetes.io/projected/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-kube-api-access-wg7z6\") pod \"maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff\" (UID: \"1950d218-889e-49d8-8bb8-11b2fc4d6ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" Sep 08 17:07:26.576673 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:26.576562 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-openshift-service-ca\") pod \"maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff\" (UID: \"1950d218-889e-49d8-8bb8-11b2fc4d6ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" Sep 08 17:07:26.624948 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:26.624896 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:26.677565 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:26.677537 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg7z6\" (UniqueName: \"kubernetes.io/projected/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-kube-api-access-wg7z6\") pod \"maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff\" (UID: \"1950d218-889e-49d8-8bb8-11b2fc4d6ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" Sep 08 17:07:26.677708 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:26.677583 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-openshift-service-ca\") pod \"maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff\" (UID: \"1950d218-889e-49d8-8bb8-11b2fc4d6ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" Sep 08 17:07:26.677708 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:26.677696 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-metrics-certs\") pod \"maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff\" (UID: \"1950d218-889e-49d8-8bb8-11b2fc4d6ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" Sep 08 17:07:26.677834 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:26.677719 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-maas-api-tls\") pod \"maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff\" (UID: \"1950d218-889e-49d8-8bb8-11b2fc4d6ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" Sep 08 17:07:26.677834 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:07:26.677823 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-serving-cert: secret "maas-api-e2e-mt-a837a8a9-serving-cert" not found Sep 08 17:07:26.677932 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:07:26.677878 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-maas-api-tls podName:1950d218-889e-49d8-8bb8-11b2fc4d6ae7 nodeName:}" failed. No retries permitted until 2026-09-08 17:07:27.177863379 +0000 UTC m=+1009.914694475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-maas-api-tls") pod "maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" (UID: "1950d218-889e-49d8-8bb8-11b2fc4d6ae7") : secret "maas-api-e2e-mt-a837a8a9-serving-cert" not found Sep 08 17:07:26.678278 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:26.678257 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-openshift-service-ca\") pod \"maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff\" (UID: \"1950d218-889e-49d8-8bb8-11b2fc4d6ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" Sep 08 17:07:26.680095 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:26.680072 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-metrics-certs\") pod \"maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff\" (UID: \"1950d218-889e-49d8-8bb8-11b2fc4d6ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" Sep 08 17:07:26.686244 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:26.686224 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg7z6\" (UniqueName: \"kubernetes.io/projected/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-kube-api-access-wg7z6\") pod \"maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff\" (UID: \"1950d218-889e-49d8-8bb8-11b2fc4d6ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" Sep 08 17:07:26.724805 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:26.724779 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:26.892182 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:26.892095 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:27.118300 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:27.118269 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:27.182327 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:27.182302 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-maas-api-tls\") pod \"maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff\" (UID: \"1950d218-889e-49d8-8bb8-11b2fc4d6ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" Sep 08 17:07:27.184616 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:27.184598 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-maas-api-tls\") pod \"maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff\" (UID: \"1950d218-889e-49d8-8bb8-11b2fc4d6ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" Sep 08 17:07:27.219464 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:27.219435 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:27.449174 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:27.449079 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" Sep 08 17:07:27.724810 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:27.724782 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff"] Sep 08 17:07:27.725772 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:07:27.725745 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1950d218_889e_49d8_8bb8_11b2fc4d6ae7.slice/crio-f99f853a4054aa7cddc910ff41afa17fac8537fb2b45b2d165149a90d4618a54 WatchSource:0}: Error finding container f99f853a4054aa7cddc910ff41afa17fac8537fb2b45b2d165149a90d4618a54: Status 404 returned error can't find the container with id f99f853a4054aa7cddc910ff41afa17fac8537fb2b45b2d165149a90d4618a54 Sep 08 17:07:27.727177 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:27.727158 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 17:07:28.415838 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:28.415796 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" event={"ID":"1950d218-889e-49d8-8bb8-11b2fc4d6ae7","Type":"ContainerStarted","Data":"f99f853a4054aa7cddc910ff41afa17fac8537fb2b45b2d165149a90d4618a54"} Sep 08 17:07:28.829505 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:28.829453 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:29.423672 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:29.423637 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" event={"ID":"1950d218-889e-49d8-8bb8-11b2fc4d6ae7","Type":"ContainerStarted","Data":"81fecb3a51d02c387140537492a87fe79e5da3a951564fc9d1e7d022b22df23a"} Sep 08 17:07:29.423883 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:29.423762 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" Sep 08 17:07:29.443834 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:29.443787 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" podStartSLOduration=2.741267504 podStartE2EDuration="3.443772079s" totalImagesPullingTime="702.504575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:07:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:07:29.441080409 +0000 UTC m=+1012.177911515" watchObservedRunningTime="2026-09-08 17:07:29.443772079 +0000 UTC m=+1012.180603194" Sep 08 17:07:35.434916 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:35.434878 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" Sep 08 17:07:48.444960 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:48.444917 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:48.736644 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:48.736556 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:49.111319 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:49.111226 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:49.239345 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:49.239310 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:49.426390 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:49.426358 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:49.504236 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:49.504201 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:49.583410 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:49.583378 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:50.260384 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:50.260345 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:50.970787 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:50.970759 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:51.092120 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:51.092083 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:51.409918 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:51.409824 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:54.656812 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:54.656772 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:57.406685 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:57.406650 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:57.511666 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:57.511636 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:58.809550 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:58.809517 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb"] Sep 08 17:07:58.809992 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:58.809755 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" podUID="90160bdd-1ef5-43d5-9559-34574b2a69dd" containerName="maas-api" containerID="cri-o://8354ce8402bd9fd93878b2164f571ccbbf1135ecbd258b0978d568fc067e9692" gracePeriod=30 Sep 08 17:07:58.844036 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:58.844011 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:58.963794 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:58.963758 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:59.065181 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.065112 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:59.068846 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.068826 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" Sep 08 17:07:59.128090 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.128055 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:59.162187 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.162163 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90160bdd-1ef5-43d5-9559-34574b2a69dd-maas-api-tls\") pod \"90160bdd-1ef5-43d5-9559-34574b2a69dd\" (UID: \"90160bdd-1ef5-43d5-9559-34574b2a69dd\") " Sep 08 17:07:59.162296 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.162221 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-462jp\" (UniqueName: \"kubernetes.io/projected/90160bdd-1ef5-43d5-9559-34574b2a69dd-kube-api-access-462jp\") pod \"90160bdd-1ef5-43d5-9559-34574b2a69dd\" (UID: \"90160bdd-1ef5-43d5-9559-34574b2a69dd\") " Sep 08 17:07:59.162296 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.162274 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90160bdd-1ef5-43d5-9559-34574b2a69dd-openshift-service-ca\") pod \"90160bdd-1ef5-43d5-9559-34574b2a69dd\" (UID: \"90160bdd-1ef5-43d5-9559-34574b2a69dd\") " Sep 08 17:07:59.162411 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.162331 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90160bdd-1ef5-43d5-9559-34574b2a69dd-metrics-certs\") pod \"90160bdd-1ef5-43d5-9559-34574b2a69dd\" (UID: \"90160bdd-1ef5-43d5-9559-34574b2a69dd\") " Sep 08 17:07:59.162759 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.162731 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90160bdd-1ef5-43d5-9559-34574b2a69dd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "90160bdd-1ef5-43d5-9559-34574b2a69dd" (UID: "90160bdd-1ef5-43d5-9559-34574b2a69dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:07:59.164349 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.164328 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90160bdd-1ef5-43d5-9559-34574b2a69dd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "90160bdd-1ef5-43d5-9559-34574b2a69dd" (UID: "90160bdd-1ef5-43d5-9559-34574b2a69dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:07:59.164473 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.164445 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90160bdd-1ef5-43d5-9559-34574b2a69dd-kube-api-access-462jp" (OuterVolumeSpecName: "kube-api-access-462jp") pod "90160bdd-1ef5-43d5-9559-34574b2a69dd" (UID: "90160bdd-1ef5-43d5-9559-34574b2a69dd"). InnerVolumeSpecName "kube-api-access-462jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:07:59.164534 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.164477 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90160bdd-1ef5-43d5-9559-34574b2a69dd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "90160bdd-1ef5-43d5-9559-34574b2a69dd" (UID: "90160bdd-1ef5-43d5-9559-34574b2a69dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:07:59.264138 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.264108 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90160bdd-1ef5-43d5-9559-34574b2a69dd-maas-api-tls\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:07:59.264138 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.264132 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-462jp\" (UniqueName: \"kubernetes.io/projected/90160bdd-1ef5-43d5-9559-34574b2a69dd-kube-api-access-462jp\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:07:59.264297 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.264144 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90160bdd-1ef5-43d5-9559-34574b2a69dd-openshift-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:07:59.264297 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.264154 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90160bdd-1ef5-43d5-9559-34574b2a69dd-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:07:59.444616 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.444588 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:07:59.601784 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.601747 2565 generic.go:356] "Generic (PLEG): container finished" podID="90160bdd-1ef5-43d5-9559-34574b2a69dd" containerID="8354ce8402bd9fd93878b2164f571ccbbf1135ecbd258b0978d568fc067e9692" exitCode=0 Sep 08 17:07:59.601975 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.601788 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" event={"ID":"90160bdd-1ef5-43d5-9559-34574b2a69dd","Type":"ContainerDied","Data":"8354ce8402bd9fd93878b2164f571ccbbf1135ecbd258b0978d568fc067e9692"} Sep 08 17:07:59.601975 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.601826 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" Sep 08 17:07:59.601975 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.601883 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb" event={"ID":"90160bdd-1ef5-43d5-9559-34574b2a69dd","Type":"ContainerDied","Data":"16dee8ac97851c127f017cd2db9d3cc816680df228584aeef8f29b1fc0af847d"} Sep 08 17:07:59.601975 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.601913 2565 scope.go:117] "RemoveContainer" containerID="8354ce8402bd9fd93878b2164f571ccbbf1135ecbd258b0978d568fc067e9692" Sep 08 17:07:59.612499 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.612480 2565 scope.go:117] "RemoveContainer" containerID="8354ce8402bd9fd93878b2164f571ccbbf1135ecbd258b0978d568fc067e9692" Sep 08 17:07:59.612766 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:07:59.612750 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8354ce8402bd9fd93878b2164f571ccbbf1135ecbd258b0978d568fc067e9692\": container with ID starting with 8354ce8402bd9fd93878b2164f571ccbbf1135ecbd258b0978d568fc067e9692 not found: ID does not exist" containerID="8354ce8402bd9fd93878b2164f571ccbbf1135ecbd258b0978d568fc067e9692" Sep 08 17:07:59.612828 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.612773 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8354ce8402bd9fd93878b2164f571ccbbf1135ecbd258b0978d568fc067e9692"} err="failed to get container status \"8354ce8402bd9fd93878b2164f571ccbbf1135ecbd258b0978d568fc067e9692\": rpc error: code = NotFound desc = could not find container \"8354ce8402bd9fd93878b2164f571ccbbf1135ecbd258b0978d568fc067e9692\": container with ID starting with 8354ce8402bd9fd93878b2164f571ccbbf1135ecbd258b0978d568fc067e9692 not found: ID does not exist" Sep 08 17:07:59.628213 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.628187 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb"] Sep 08 17:07:59.633516 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.633484 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2746ad-684f6965df-2rqcb"] Sep 08 17:07:59.760087 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:07:59.760015 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90160bdd-1ef5-43d5-9559-34574b2a69dd" path="/var/lib/kubelet/pods/90160bdd-1ef5-43d5-9559-34574b2a69dd/volumes" Sep 08 17:08:05.338731 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:05.338696 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:05.441698 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:05.441662 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:05.594457 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:05.594319 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:06.050523 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:06.050469 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:07.852837 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:07.852801 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:08.421558 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:08.421526 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff"] Sep 08 17:08:08.421778 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:08.421753 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" podUID="1950d218-889e-49d8-8bb8-11b2fc4d6ae7" containerName="maas-api" containerID="cri-o://81fecb3a51d02c387140537492a87fe79e5da3a951564fc9d1e7d022b22df23a" gracePeriod=30 Sep 08 17:08:08.451937 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:08.451904 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:08.554256 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:08.553740 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:08.653289 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:08.653249 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:08.658324 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:08.658299 2565 generic.go:356] "Generic (PLEG): container finished" podID="1950d218-889e-49d8-8bb8-11b2fc4d6ae7" containerID="81fecb3a51d02c387140537492a87fe79e5da3a951564fc9d1e7d022b22df23a" exitCode=0 Sep 08 17:08:08.658460 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:08.658368 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" event={"ID":"1950d218-889e-49d8-8bb8-11b2fc4d6ae7","Type":"ContainerDied","Data":"81fecb3a51d02c387140537492a87fe79e5da3a951564fc9d1e7d022b22df23a"} Sep 08 17:08:08.675838 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:08.675788 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" Sep 08 17:08:08.738506 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:08.738461 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-openshift-service-ca\") pod \"1950d218-889e-49d8-8bb8-11b2fc4d6ae7\" (UID: \"1950d218-889e-49d8-8bb8-11b2fc4d6ae7\") " Sep 08 17:08:08.738666 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:08.738529 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-metrics-certs\") pod \"1950d218-889e-49d8-8bb8-11b2fc4d6ae7\" (UID: \"1950d218-889e-49d8-8bb8-11b2fc4d6ae7\") " Sep 08 17:08:08.738666 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:08.738556 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-maas-api-tls\") pod \"1950d218-889e-49d8-8bb8-11b2fc4d6ae7\" (UID: \"1950d218-889e-49d8-8bb8-11b2fc4d6ae7\") " Sep 08 17:08:08.738916 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:08.738891 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1950d218-889e-49d8-8bb8-11b2fc4d6ae7" (UID: "1950d218-889e-49d8-8bb8-11b2fc4d6ae7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:08:08.740516 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:08.740486 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1950d218-889e-49d8-8bb8-11b2fc4d6ae7" (UID: "1950d218-889e-49d8-8bb8-11b2fc4d6ae7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:08:08.740924 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:08.740899 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1950d218-889e-49d8-8bb8-11b2fc4d6ae7" (UID: "1950d218-889e-49d8-8bb8-11b2fc4d6ae7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:08:08.753512 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:08.753480 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:08.839076 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:08.839050 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wg7z6\" (UniqueName: \"kubernetes.io/projected/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-kube-api-access-wg7z6\") pod \"1950d218-889e-49d8-8bb8-11b2fc4d6ae7\" (UID: \"1950d218-889e-49d8-8bb8-11b2fc4d6ae7\") " Sep 08 17:08:08.839246 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:08.839233 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-openshift-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:08:08.839297 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:08.839248 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:08:08.839297 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:08.839257 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-maas-api-tls\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:08:08.841099 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:08.841065 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-kube-api-access-wg7z6" (OuterVolumeSpecName: "kube-api-access-wg7z6") pod "1950d218-889e-49d8-8bb8-11b2fc4d6ae7" (UID: "1950d218-889e-49d8-8bb8-11b2fc4d6ae7"). InnerVolumeSpecName "kube-api-access-wg7z6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:08:08.939837 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:08.939775 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg7z6\" (UniqueName: \"kubernetes.io/projected/1950d218-889e-49d8-8bb8-11b2fc4d6ae7-kube-api-access-wg7z6\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:08:08.989234 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:08.989193 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:09.666419 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:09.666394 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" Sep 08 17:08:09.666619 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:09.666453 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff" event={"ID":"1950d218-889e-49d8-8bb8-11b2fc4d6ae7","Type":"ContainerDied","Data":"f99f853a4054aa7cddc910ff41afa17fac8537fb2b45b2d165149a90d4618a54"} Sep 08 17:08:09.666619 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:09.666530 2565 scope.go:117] "RemoveContainer" containerID="81fecb3a51d02c387140537492a87fe79e5da3a951564fc9d1e7d022b22df23a" Sep 08 17:08:09.688232 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:09.688181 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff"] Sep 08 17:08:09.693064 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:09.693045 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a837a8a9-64b8d7f74c-gdtff"] Sep 08 17:08:09.760547 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:09.760519 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1950d218-889e-49d8-8bb8-11b2fc4d6ae7" path="/var/lib/kubelet/pods/1950d218-889e-49d8-8bb8-11b2fc4d6ae7/volumes" Sep 08 17:08:14.799930 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:14.799888 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:14.904708 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:14.904679 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:15.090619 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:15.090532 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:15.462997 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:15.462963 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:15.571148 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:15.571114 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:15.676972 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:15.676944 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:16.133836 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:16.133743 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:16.714621 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:16.714579 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj"] Sep 08 17:08:16.714834 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:16.714803 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" podUID="73db64db-594c-4f41-b959-645623e61c11" containerName="maas-api" containerID="cri-o://3640a5bb16a24b4854e95913eda142f89fb990e253edf84d104b107c646d4ce4" gracePeriod=30 Sep 08 17:08:16.748570 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:16.748459 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:16.849501 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:16.849361 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:16.944772 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:16.944711 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:16.980696 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:16.980671 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" Sep 08 17:08:17.004851 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.004823 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmwbk\" (UniqueName: \"kubernetes.io/projected/73db64db-594c-4f41-b959-645623e61c11-kube-api-access-pmwbk\") pod \"73db64db-594c-4f41-b959-645623e61c11\" (UID: \"73db64db-594c-4f41-b959-645623e61c11\") " Sep 08 17:08:17.004988 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.004881 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73db64db-594c-4f41-b959-645623e61c11-openshift-service-ca\") pod \"73db64db-594c-4f41-b959-645623e61c11\" (UID: \"73db64db-594c-4f41-b959-645623e61c11\") " Sep 08 17:08:17.004988 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.004917 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73db64db-594c-4f41-b959-645623e61c11-maas-api-tls\") pod \"73db64db-594c-4f41-b959-645623e61c11\" (UID: \"73db64db-594c-4f41-b959-645623e61c11\") " Sep 08 17:08:17.004988 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.004960 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73db64db-594c-4f41-b959-645623e61c11-metrics-certs\") pod \"73db64db-594c-4f41-b959-645623e61c11\" (UID: \"73db64db-594c-4f41-b959-645623e61c11\") " Sep 08 17:08:17.005325 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.005293 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73db64db-594c-4f41-b959-645623e61c11-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "73db64db-594c-4f41-b959-645623e61c11" (UID: "73db64db-594c-4f41-b959-645623e61c11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:08:17.007234 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.007205 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73db64db-594c-4f41-b959-645623e61c11-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "73db64db-594c-4f41-b959-645623e61c11" (UID: "73db64db-594c-4f41-b959-645623e61c11"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:08:17.007357 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.007342 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73db64db-594c-4f41-b959-645623e61c11-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "73db64db-594c-4f41-b959-645623e61c11" (UID: "73db64db-594c-4f41-b959-645623e61c11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:08:17.007560 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.007543 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73db64db-594c-4f41-b959-645623e61c11-kube-api-access-pmwbk" (OuterVolumeSpecName: "kube-api-access-pmwbk") pod "73db64db-594c-4f41-b959-645623e61c11" (UID: "73db64db-594c-4f41-b959-645623e61c11"). InnerVolumeSpecName "kube-api-access-pmwbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:08:17.106129 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.106091 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73db64db-594c-4f41-b959-645623e61c11-openshift-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:08:17.106129 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.106128 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73db64db-594c-4f41-b959-645623e61c11-maas-api-tls\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:08:17.106129 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.106138 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73db64db-594c-4f41-b959-645623e61c11-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:08:17.106336 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.106147 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmwbk\" (UniqueName: \"kubernetes.io/projected/73db64db-594c-4f41-b959-645623e61c11-kube-api-access-pmwbk\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:08:17.162119 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.162089 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:17.320948 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.320840 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:17.715597 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.715563 2565 generic.go:356] "Generic (PLEG): container finished" podID="73db64db-594c-4f41-b959-645623e61c11" containerID="3640a5bb16a24b4854e95913eda142f89fb990e253edf84d104b107c646d4ce4" exitCode=0 Sep 08 17:08:17.715767 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.715611 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" Sep 08 17:08:17.715767 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.715653 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" event={"ID":"73db64db-594c-4f41-b959-645623e61c11","Type":"ContainerDied","Data":"3640a5bb16a24b4854e95913eda142f89fb990e253edf84d104b107c646d4ce4"} Sep 08 17:08:17.715767 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.715707 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj" event={"ID":"73db64db-594c-4f41-b959-645623e61c11","Type":"ContainerDied","Data":"495333cc953ef1ee728558a5bf31a6b1a3efff324699f313a88f09fc6e4b8201"} Sep 08 17:08:17.715767 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.715729 2565 scope.go:117] "RemoveContainer" containerID="3640a5bb16a24b4854e95913eda142f89fb990e253edf84d104b107c646d4ce4" Sep 08 17:08:17.733291 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.733271 2565 scope.go:117] "RemoveContainer" containerID="3640a5bb16a24b4854e95913eda142f89fb990e253edf84d104b107c646d4ce4" Sep 08 17:08:17.733580 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:08:17.733557 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3640a5bb16a24b4854e95913eda142f89fb990e253edf84d104b107c646d4ce4\": container with ID starting with 3640a5bb16a24b4854e95913eda142f89fb990e253edf84d104b107c646d4ce4 not found: ID does not exist" containerID="3640a5bb16a24b4854e95913eda142f89fb990e253edf84d104b107c646d4ce4" Sep 08 17:08:17.733658 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.733591 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3640a5bb16a24b4854e95913eda142f89fb990e253edf84d104b107c646d4ce4"} err="failed to get container status \"3640a5bb16a24b4854e95913eda142f89fb990e253edf84d104b107c646d4ce4\": rpc error: code = NotFound desc = could not find container \"3640a5bb16a24b4854e95913eda142f89fb990e253edf84d104b107c646d4ce4\": container with ID starting with 3640a5bb16a24b4854e95913eda142f89fb990e253edf84d104b107c646d4ce4 not found: ID does not exist" Sep 08 17:08:17.743969 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.743942 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj"] Sep 08 17:08:17.750731 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.750706 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7cab-d89c55589-fdglj"] Sep 08 17:08:17.762320 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:17.762286 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73db64db-594c-4f41-b959-645623e61c11" path="/var/lib/kubelet/pods/73db64db-594c-4f41-b959-645623e61c11/volumes" Sep 08 17:08:21.541504 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:21.541464 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:21.641627 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:21.641585 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:21.741745 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:21.741715 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:21.999235 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:21.999194 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:23.580628 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:23.580592 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:23.681240 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:23.681207 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:23.832298 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:23.832195 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:24.369950 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:24.369917 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:24.800923 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:24.800886 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:24.914871 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:24.914843 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:27.626984 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:27.626947 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:27.763293 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:27.763255 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:28.220257 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.220221 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:28.294268 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.294233 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:28.735669 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.735636 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z"] Sep 08 17:08:28.736165 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.736028 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90160bdd-1ef5-43d5-9559-34574b2a69dd" containerName="maas-api" Sep 08 17:08:28.736165 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.736053 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="90160bdd-1ef5-43d5-9559-34574b2a69dd" containerName="maas-api" Sep 08 17:08:28.736165 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.736061 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1950d218-889e-49d8-8bb8-11b2fc4d6ae7" containerName="maas-api" Sep 08 17:08:28.736165 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.736066 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1950d218-889e-49d8-8bb8-11b2fc4d6ae7" containerName="maas-api" Sep 08 17:08:28.736165 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.736078 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73db64db-594c-4f41-b959-645623e61c11" containerName="maas-api" Sep 08 17:08:28.736165 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.736084 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="73db64db-594c-4f41-b959-645623e61c11" containerName="maas-api" Sep 08 17:08:28.736165 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.736144 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="73db64db-594c-4f41-b959-645623e61c11" containerName="maas-api" Sep 08 17:08:28.736165 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.736152 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="90160bdd-1ef5-43d5-9559-34574b2a69dd" containerName="maas-api" Sep 08 17:08:28.736165 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.736161 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1950d218-889e-49d8-8bb8-11b2fc4d6ae7" containerName="maas-api" Sep 08 17:08:28.740756 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.740729 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" Sep 08 17:08:28.743120 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.743101 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-c05c9f-serving-cert\"" Sep 08 17:08:28.758290 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.758252 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z"] Sep 08 17:08:28.786994 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.786946 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:28.804745 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.804721 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nctld\" (UniqueName: \"kubernetes.io/projected/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-kube-api-access-nctld\") pod \"maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z\" (UID: \"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" Sep 08 17:08:28.804883 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.804782 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z\" (UID: \"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" Sep 08 17:08:28.804883 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.804806 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z\" (UID: \"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" Sep 08 17:08:28.804883 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.804827 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z\" (UID: \"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" Sep 08 17:08:28.873163 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.873130 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:28.905323 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.905289 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nctld\" (UniqueName: \"kubernetes.io/projected/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-kube-api-access-nctld\") pod \"maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z\" (UID: \"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" Sep 08 17:08:28.905552 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.905348 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z\" (UID: \"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" Sep 08 17:08:28.905552 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.905382 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z\" (UID: \"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" Sep 08 17:08:28.905552 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.905405 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z\" (UID: \"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" Sep 08 17:08:28.906131 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.906100 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z\" (UID: \"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" Sep 08 17:08:28.907827 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.907804 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z\" (UID: \"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" Sep 08 17:08:28.907910 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.907884 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z\" (UID: \"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" Sep 08 17:08:28.918884 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:28.918858 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nctld\" (UniqueName: \"kubernetes.io/projected/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-kube-api-access-nctld\") pod \"maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z\" (UID: \"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" Sep 08 17:08:29.057610 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:29.057536 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" Sep 08 17:08:29.167930 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:29.167898 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:29.334969 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:29.334935 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z"] Sep 08 17:08:29.337357 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:08:29.337319 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1409c50_bcbc_48c7_b7ab_dba7bed9ab0d.slice/crio-b97a1a4e7fdb7373320ccdfbe336ce09b71a911ddcc3175e5964db7caa9c7193 WatchSource:0}: Error finding container b97a1a4e7fdb7373320ccdfbe336ce09b71a911ddcc3175e5964db7caa9c7193: Status 404 returned error can't find the container with id b97a1a4e7fdb7373320ccdfbe336ce09b71a911ddcc3175e5964db7caa9c7193 Sep 08 17:08:29.419934 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:29.419901 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:29.518055 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:29.518024 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:29.773059 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:29.773023 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:29.786772 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:29.786743 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" event={"ID":"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d","Type":"ContainerStarted","Data":"b97a1a4e7fdb7373320ccdfbe336ce09b71a911ddcc3175e5964db7caa9c7193"} Sep 08 17:08:30.793957 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:30.793920 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" event={"ID":"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d","Type":"ContainerStarted","Data":"7066f364fde421972bf7e77d622a3473a493a3103ae506cac82e25b38402d2a0"} Sep 08 17:08:30.794409 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:30.793988 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" Sep 08 17:08:30.814523 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:30.814474 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" podStartSLOduration=2.32314619 podStartE2EDuration="2.814457679s" totalImagesPullingTime="491.311489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:08:30.812238963 +0000 UTC m=+1073.549070078" watchObservedRunningTime="2026-09-08 17:08:30.814457679 +0000 UTC m=+1073.551288796" Sep 08 17:08:31.699362 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:31.699318 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:32.741634 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:32.741580 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:32.835803 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:32.835761 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:33.821642 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:33.821606 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg"] Sep 08 17:08:33.825667 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:33.825642 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" Sep 08 17:08:33.828119 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:33.828097 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-ec96ff-serving-cert\"" Sep 08 17:08:33.836990 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:33.836958 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg"] Sep 08 17:08:33.858774 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:33.858741 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:33.948871 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:33.948830 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57ed1a95-f79c-4242-a5f2-45eafca7b893-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg\" (UID: \"57ed1a95-f79c-4242-a5f2-45eafca7b893\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" Sep 08 17:08:33.949095 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:33.948912 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xts9\" (UniqueName: \"kubernetes.io/projected/57ed1a95-f79c-4242-a5f2-45eafca7b893-kube-api-access-5xts9\") pod \"maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg\" (UID: \"57ed1a95-f79c-4242-a5f2-45eafca7b893\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" Sep 08 17:08:33.949095 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:33.948956 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ed1a95-f79c-4242-a5f2-45eafca7b893-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg\" (UID: \"57ed1a95-f79c-4242-a5f2-45eafca7b893\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" Sep 08 17:08:33.949095 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:33.948980 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ed1a95-f79c-4242-a5f2-45eafca7b893-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg\" (UID: \"57ed1a95-f79c-4242-a5f2-45eafca7b893\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" Sep 08 17:08:33.958227 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:33.958196 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:34.050413 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:34.050363 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57ed1a95-f79c-4242-a5f2-45eafca7b893-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg\" (UID: \"57ed1a95-f79c-4242-a5f2-45eafca7b893\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" Sep 08 17:08:34.050640 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:34.050477 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xts9\" (UniqueName: \"kubernetes.io/projected/57ed1a95-f79c-4242-a5f2-45eafca7b893-kube-api-access-5xts9\") pod \"maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg\" (UID: \"57ed1a95-f79c-4242-a5f2-45eafca7b893\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" Sep 08 17:08:34.050640 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:34.050517 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ed1a95-f79c-4242-a5f2-45eafca7b893-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg\" (UID: \"57ed1a95-f79c-4242-a5f2-45eafca7b893\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" Sep 08 17:08:34.050640 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:34.050542 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ed1a95-f79c-4242-a5f2-45eafca7b893-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg\" (UID: \"57ed1a95-f79c-4242-a5f2-45eafca7b893\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" Sep 08 17:08:34.051302 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:34.051274 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ed1a95-f79c-4242-a5f2-45eafca7b893-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg\" (UID: \"57ed1a95-f79c-4242-a5f2-45eafca7b893\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" Sep 08 17:08:34.053251 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:34.053216 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ed1a95-f79c-4242-a5f2-45eafca7b893-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg\" (UID: \"57ed1a95-f79c-4242-a5f2-45eafca7b893\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" Sep 08 17:08:34.053374 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:34.053313 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57ed1a95-f79c-4242-a5f2-45eafca7b893-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg\" (UID: \"57ed1a95-f79c-4242-a5f2-45eafca7b893\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" Sep 08 17:08:34.058869 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:34.058838 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:34.058983 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:34.058921 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xts9\" (UniqueName: \"kubernetes.io/projected/57ed1a95-f79c-4242-a5f2-45eafca7b893-kube-api-access-5xts9\") pod \"maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg\" (UID: \"57ed1a95-f79c-4242-a5f2-45eafca7b893\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" Sep 08 17:08:34.140012 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:34.139922 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" Sep 08 17:08:34.423999 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:34.423965 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg"] Sep 08 17:08:34.424752 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:08:34.424719 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ed1a95_f79c_4242_a5f2_45eafca7b893.slice/crio-6f6698fe35a7d77057267945791e2d524b52696fe47533dd3db5cc2246b3e019 WatchSource:0}: Error finding container 6f6698fe35a7d77057267945791e2d524b52696fe47533dd3db5cc2246b3e019: Status 404 returned error can't find the container with id 6f6698fe35a7d77057267945791e2d524b52696fe47533dd3db5cc2246b3e019 Sep 08 17:08:34.453276 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:34.453240 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:34.555653 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:34.555614 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:34.820103 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:34.820067 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" event={"ID":"57ed1a95-f79c-4242-a5f2-45eafca7b893","Type":"ContainerStarted","Data":"4fbbcfc97a37ecdc1f5e683edfb63c2c226c16b6f7364e00be0ab4c4d566fe77"} Sep 08 17:08:34.820293 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:34.820131 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" event={"ID":"57ed1a95-f79c-4242-a5f2-45eafca7b893","Type":"ContainerStarted","Data":"6f6698fe35a7d77057267945791e2d524b52696fe47533dd3db5cc2246b3e019"} Sep 08 17:08:34.820293 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:34.820245 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" Sep 08 17:08:34.839675 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:34.839624 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" podStartSLOduration=1.5441448 podStartE2EDuration="1.839608122s" totalImagesPullingTime="295.463322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:08:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:08:34.837479266 +0000 UTC m=+1077.574310385" watchObservedRunningTime="2026-09-08 17:08:34.839608122 +0000 UTC m=+1077.576439239" Sep 08 17:08:36.803867 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:36.803839 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" Sep 08 17:08:40.831941 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:40.831910 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" Sep 08 17:08:49.891685 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:49.891651 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:52.553771 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:52.553730 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:52.874751 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:52.874665 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:53.188280 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:53.188252 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:53.425248 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:53.425216 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:54.875670 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:54.875636 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:55.897166 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:55.897127 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:56.017683 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:56.017654 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:57.445993 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:57.445957 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6"] Sep 08 17:08:57.449939 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:57.449917 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" Sep 08 17:08:57.456210 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:57.456186 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-338600-serving-cert\"" Sep 08 17:08:57.465584 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:57.465562 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6"] Sep 08 17:08:57.467968 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:57.467944 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0019913a-b578-414a-8e5c-d03cd746c1d0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6\" (UID: \"0019913a-b578-414a-8e5c-d03cd746c1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" Sep 08 17:08:57.468085 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:57.468000 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbxnz\" (UniqueName: \"kubernetes.io/projected/0019913a-b578-414a-8e5c-d03cd746c1d0-kube-api-access-cbxnz\") pod \"maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6\" (UID: \"0019913a-b578-414a-8e5c-d03cd746c1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" Sep 08 17:08:57.468143 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:57.468120 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0019913a-b578-414a-8e5c-d03cd746c1d0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6\" (UID: \"0019913a-b578-414a-8e5c-d03cd746c1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" Sep 08 17:08:57.468199 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:57.468175 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0019913a-b578-414a-8e5c-d03cd746c1d0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6\" (UID: \"0019913a-b578-414a-8e5c-d03cd746c1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" Sep 08 17:08:57.528551 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:57.528517 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:57.569493 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:57.569467 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0019913a-b578-414a-8e5c-d03cd746c1d0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6\" (UID: \"0019913a-b578-414a-8e5c-d03cd746c1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" Sep 08 17:08:57.569663 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:57.569517 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbxnz\" (UniqueName: \"kubernetes.io/projected/0019913a-b578-414a-8e5c-d03cd746c1d0-kube-api-access-cbxnz\") pod \"maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6\" (UID: \"0019913a-b578-414a-8e5c-d03cd746c1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" Sep 08 17:08:57.569663 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:57.569581 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0019913a-b578-414a-8e5c-d03cd746c1d0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6\" (UID: \"0019913a-b578-414a-8e5c-d03cd746c1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" Sep 08 17:08:57.569663 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:08:57.569617 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-serving-cert: secret "maas-api-e2e-shared-b-w3-338600-serving-cert" not found Sep 08 17:08:57.569831 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:08:57.569688 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0019913a-b578-414a-8e5c-d03cd746c1d0-maas-api-tls podName:0019913a-b578-414a-8e5c-d03cd746c1d0 nodeName:}" failed. No retries permitted until 2026-09-08 17:08:58.069671266 +0000 UTC m=+1100.806502364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0019913a-b578-414a-8e5c-d03cd746c1d0-maas-api-tls") pod "maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" (UID: "0019913a-b578-414a-8e5c-d03cd746c1d0") : secret "maas-api-e2e-shared-b-w3-338600-serving-cert" not found Sep 08 17:08:57.569831 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:57.569623 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0019913a-b578-414a-8e5c-d03cd746c1d0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6\" (UID: \"0019913a-b578-414a-8e5c-d03cd746c1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" Sep 08 17:08:57.570286 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:57.570262 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0019913a-b578-414a-8e5c-d03cd746c1d0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6\" (UID: \"0019913a-b578-414a-8e5c-d03cd746c1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" Sep 08 17:08:57.572015 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:57.571997 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0019913a-b578-414a-8e5c-d03cd746c1d0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6\" (UID: \"0019913a-b578-414a-8e5c-d03cd746c1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" Sep 08 17:08:57.595771 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:57.595748 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbxnz\" (UniqueName: \"kubernetes.io/projected/0019913a-b578-414a-8e5c-d03cd746c1d0-kube-api-access-cbxnz\") pod \"maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6\" (UID: \"0019913a-b578-414a-8e5c-d03cd746c1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" Sep 08 17:08:57.657528 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:57.657466 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:57.919592 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:57.919557 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:58.033411 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:58.033362 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:58.072717 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:58.072678 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0019913a-b578-414a-8e5c-d03cd746c1d0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6\" (UID: \"0019913a-b578-414a-8e5c-d03cd746c1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" Sep 08 17:08:58.075181 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:58.075151 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0019913a-b578-414a-8e5c-d03cd746c1d0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6\" (UID: \"0019913a-b578-414a-8e5c-d03cd746c1d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" Sep 08 17:08:58.301018 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:58.300923 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:58.360863 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:58.360829 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" Sep 08 17:08:58.451785 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:58.451749 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:58.640406 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:58.640323 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:58.654937 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:58.654910 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6"] Sep 08 17:08:58.656254 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:08:58.656220 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0019913a_b578_414a_8e5c_d03cd746c1d0.slice/crio-5c78c2b32f8ac9019449527c8ae3a84a1d0a2a09a985a7262d5dca935f99e128 WatchSource:0}: Error finding container 5c78c2b32f8ac9019449527c8ae3a84a1d0a2a09a985a7262d5dca935f99e128: Status 404 returned error can't find the container with id 5c78c2b32f8ac9019449527c8ae3a84a1d0a2a09a985a7262d5dca935f99e128 Sep 08 17:08:58.959202 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:58.959165 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" event={"ID":"0019913a-b578-414a-8e5c-d03cd746c1d0","Type":"ContainerStarted","Data":"5c78c2b32f8ac9019449527c8ae3a84a1d0a2a09a985a7262d5dca935f99e128"} Sep 08 17:08:59.212372 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:59.212286 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:08:59.967233 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:59.967190 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" event={"ID":"0019913a-b578-414a-8e5c-d03cd746c1d0","Type":"ContainerStarted","Data":"5540178d0dd3793be268b09fd548118d5c3f67e60e230c42522f90da0cb6ed53"} Sep 08 17:08:59.967708 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:59.967256 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" Sep 08 17:08:59.997773 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:08:59.997717 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" podStartSLOduration=2.404542941 podStartE2EDuration="2.997701452s" totalImagesPullingTime="593.158511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:08:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:08:59.995450609 +0000 UTC m=+1102.732281727" watchObservedRunningTime="2026-09-08 17:08:59.997701452 +0000 UTC m=+1102.734532568" Sep 08 17:09:01.567766 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:01.567725 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:01.671772 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:01.671720 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:02.086832 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:02.086792 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:02.277972 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:02.277934 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:03.252093 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:03.252046 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m"] Sep 08 17:09:03.257388 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:03.257361 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" Sep 08 17:09:03.260414 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:03.260389 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-d7359b-serving-cert\"" Sep 08 17:09:03.270777 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:03.270754 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m"] Sep 08 17:09:03.288841 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:03.288815 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:03.322494 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:03.322459 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7tpd\" (UniqueName: \"kubernetes.io/projected/bacb3d2c-1b29-4413-a5db-81288da98dc5-kube-api-access-s7tpd\") pod \"maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m\" (UID: \"bacb3d2c-1b29-4413-a5db-81288da98dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" Sep 08 17:09:03.322645 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:03.322507 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bacb3d2c-1b29-4413-a5db-81288da98dc5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m\" (UID: \"bacb3d2c-1b29-4413-a5db-81288da98dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" Sep 08 17:09:03.322645 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:03.322588 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bacb3d2c-1b29-4413-a5db-81288da98dc5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m\" (UID: \"bacb3d2c-1b29-4413-a5db-81288da98dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" Sep 08 17:09:03.322645 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:03.322622 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bacb3d2c-1b29-4413-a5db-81288da98dc5-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m\" (UID: \"bacb3d2c-1b29-4413-a5db-81288da98dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" Sep 08 17:09:03.375221 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:03.375185 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:03.423554 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:03.423516 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bacb3d2c-1b29-4413-a5db-81288da98dc5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m\" (UID: \"bacb3d2c-1b29-4413-a5db-81288da98dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" Sep 08 17:09:03.423734 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:03.423566 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bacb3d2c-1b29-4413-a5db-81288da98dc5-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m\" (UID: \"bacb3d2c-1b29-4413-a5db-81288da98dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" Sep 08 17:09:03.423734 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:03.423663 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7tpd\" (UniqueName: \"kubernetes.io/projected/bacb3d2c-1b29-4413-a5db-81288da98dc5-kube-api-access-s7tpd\") pod \"maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m\" (UID: \"bacb3d2c-1b29-4413-a5db-81288da98dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" Sep 08 17:09:03.423734 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:03.423698 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-serving-cert: secret "maas-api-e2e-shared-b-w2-d7359b-serving-cert" not found Sep 08 17:09:03.423734 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:03.423708 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bacb3d2c-1b29-4413-a5db-81288da98dc5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m\" (UID: \"bacb3d2c-1b29-4413-a5db-81288da98dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" Sep 08 17:09:03.423923 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:03.423766 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bacb3d2c-1b29-4413-a5db-81288da98dc5-maas-api-tls podName:bacb3d2c-1b29-4413-a5db-81288da98dc5 nodeName:}" failed. No retries permitted until 2026-09-08 17:09:03.92374298 +0000 UTC m=+1106.660574076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bacb3d2c-1b29-4413-a5db-81288da98dc5-maas-api-tls") pod "maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" (UID: "bacb3d2c-1b29-4413-a5db-81288da98dc5") : secret "maas-api-e2e-shared-b-w2-d7359b-serving-cert" not found Sep 08 17:09:03.424540 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:03.424514 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bacb3d2c-1b29-4413-a5db-81288da98dc5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m\" (UID: \"bacb3d2c-1b29-4413-a5db-81288da98dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" Sep 08 17:09:03.426247 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:03.426218 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bacb3d2c-1b29-4413-a5db-81288da98dc5-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m\" (UID: \"bacb3d2c-1b29-4413-a5db-81288da98dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" Sep 08 17:09:03.438197 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:03.438173 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7tpd\" (UniqueName: \"kubernetes.io/projected/bacb3d2c-1b29-4413-a5db-81288da98dc5-kube-api-access-s7tpd\") pod \"maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m\" (UID: \"bacb3d2c-1b29-4413-a5db-81288da98dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" Sep 08 17:09:03.797597 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:03.797562 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:03.930320 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:03.930278 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bacb3d2c-1b29-4413-a5db-81288da98dc5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m\" (UID: \"bacb3d2c-1b29-4413-a5db-81288da98dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" Sep 08 17:09:03.933709 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:03.933678 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bacb3d2c-1b29-4413-a5db-81288da98dc5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m\" (UID: \"bacb3d2c-1b29-4413-a5db-81288da98dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" Sep 08 17:09:03.977804 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:03.977768 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:04.170331 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:04.170302 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" Sep 08 17:09:04.371715 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:04.371676 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:04.460109 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:04.460083 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m"] Sep 08 17:09:04.462957 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:09:04.462926 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbacb3d2c_1b29_4413_a5db_81288da98dc5.slice/crio-3b9ae4f92a3839d54dc43655368eae8de4f14e7e36bc9b9301073e777ac04460 WatchSource:0}: Error finding container 3b9ae4f92a3839d54dc43655368eae8de4f14e7e36bc9b9301073e777ac04460: Status 404 returned error can't find the container with id 3b9ae4f92a3839d54dc43655368eae8de4f14e7e36bc9b9301073e777ac04460 Sep 08 17:09:05.003000 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:05.002969 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" event={"ID":"bacb3d2c-1b29-4413-a5db-81288da98dc5","Type":"ContainerStarted","Data":"3b9ae4f92a3839d54dc43655368eae8de4f14e7e36bc9b9301073e777ac04460"} Sep 08 17:09:05.979120 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:05.979086 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" Sep 08 17:09:06.013980 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:06.013947 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" event={"ID":"bacb3d2c-1b29-4413-a5db-81288da98dc5","Type":"ContainerStarted","Data":"e965505b888ed8e752b53fba8992c23d89bf217b8858f197e507e7e6dafd7cb3"} Sep 08 17:09:06.014155 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:06.014006 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" Sep 08 17:09:06.030376 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:06.030318 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" podStartSLOduration=2.5703891690000003 podStartE2EDuration="3.030295354s" totalImagesPullingTime="459.906185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:09:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:09:06.029221971 +0000 UTC m=+1108.766053086" watchObservedRunningTime="2026-09-08 17:09:06.030295354 +0000 UTC m=+1108.767126471" Sep 08 17:09:12.024919 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:12.024887 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" Sep 08 17:09:12.388012 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:12.387911 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:19.056315 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:19.056278 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:20.819473 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:20.819414 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:22.016112 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:22.016079 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:24.489980 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:24.489924 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:25.676192 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:25.676150 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:26.884695 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:26.884652 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:27.604353 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:27.604312 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7829-a87dbc"] Sep 08 17:09:27.608233 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:27.608217 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7829-a87dbc" Sep 08 17:09:27.610635 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:27.610612 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v8zrb\"" Sep 08 17:09:27.617835 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:27.617803 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7829-a87dbc"] Sep 08 17:09:27.749170 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:27.749123 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqsvp\" (UniqueName: \"kubernetes.io/projected/3bfd2a2b-d4d6-4456-a4f1-9173b2df4740-kube-api-access-lqsvp\") pod \"test-curl-7829-a87dbc\" (UID: \"3bfd2a2b-d4d6-4456-a4f1-9173b2df4740\") " pod="opendatahub/test-curl-7829-a87dbc" Sep 08 17:09:27.849722 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:27.849680 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqsvp\" (UniqueName: \"kubernetes.io/projected/3bfd2a2b-d4d6-4456-a4f1-9173b2df4740-kube-api-access-lqsvp\") pod \"test-curl-7829-a87dbc\" (UID: \"3bfd2a2b-d4d6-4456-a4f1-9173b2df4740\") " pod="opendatahub/test-curl-7829-a87dbc" Sep 08 17:09:27.862666 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:27.862592 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqsvp\" (UniqueName: \"kubernetes.io/projected/3bfd2a2b-d4d6-4456-a4f1-9173b2df4740-kube-api-access-lqsvp\") pod \"test-curl-7829-a87dbc\" (UID: \"3bfd2a2b-d4d6-4456-a4f1-9173b2df4740\") " pod="opendatahub/test-curl-7829-a87dbc" Sep 08 17:09:27.918195 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:27.918160 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7829-a87dbc" Sep 08 17:09:28.195386 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:28.192194 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7829-a87dbc"] Sep 08 17:09:29.154797 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:29.154759 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7829-a87dbc" event={"ID":"3bfd2a2b-d4d6-4456-a4f1-9173b2df4740","Type":"ContainerStarted","Data":"e472b2ac19d536fac821204e364f71e337aecd2d936a2ad09a16985ff9643722"} Sep 08 17:09:29.155209 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:29.154808 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7829-a87dbc" event={"ID":"3bfd2a2b-d4d6-4456-a4f1-9173b2df4740","Type":"ContainerStarted","Data":"5e160f83c465c8f9642e8c1e842ea27ac2ab1fe5de186862d51c66e150a2eaa2"} Sep 08 17:09:29.168030 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:29.167975 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7829-a87dbc" podStartSLOduration=2.167955973 podStartE2EDuration="2.167955973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:09:29.167413914 +0000 UTC m=+1131.904245029" watchObservedRunningTime="2026-09-08 17:09:29.167955973 +0000 UTC m=+1131.904787090" Sep 08 17:09:29.751498 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:29.751457 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7829-a87dbc"] Sep 08 17:09:29.761412 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:29.761377 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7829-a87dbc"] Sep 08 17:09:29.875311 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:29.875234 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7829-1623ae"] Sep 08 17:09:29.875726 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:29.875711 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bfd2a2b-d4d6-4456-a4f1-9173b2df4740" containerName="test-curl-7829-a87dbc" Sep 08 17:09:29.875790 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:29.875730 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bfd2a2b-d4d6-4456-a4f1-9173b2df4740" containerName="test-curl-7829-a87dbc" Sep 08 17:09:29.875829 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:29.875801 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bfd2a2b-d4d6-4456-a4f1-9173b2df4740" containerName="test-curl-7829-a87dbc" Sep 08 17:09:29.879052 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:29.879032 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7829-1623ae" Sep 08 17:09:29.885601 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:29.885579 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7829-1623ae"] Sep 08 17:09:29.969660 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:29.969624 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h45jf\" (UniqueName: \"kubernetes.io/projected/d1ead5a1-6023-4194-b569-3a93d9f88447-kube-api-access-h45jf\") pod \"test-curl-7829-1623ae\" (UID: \"d1ead5a1-6023-4194-b569-3a93d9f88447\") " pod="opendatahub/test-curl-7829-1623ae" Sep 08 17:09:30.071035 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:30.070993 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h45jf\" (UniqueName: \"kubernetes.io/projected/d1ead5a1-6023-4194-b569-3a93d9f88447-kube-api-access-h45jf\") pod \"test-curl-7829-1623ae\" (UID: \"d1ead5a1-6023-4194-b569-3a93d9f88447\") " pod="opendatahub/test-curl-7829-1623ae" Sep 08 17:09:30.080386 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:30.080360 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h45jf\" (UniqueName: \"kubernetes.io/projected/d1ead5a1-6023-4194-b569-3a93d9f88447-kube-api-access-h45jf\") pod \"test-curl-7829-1623ae\" (UID: \"d1ead5a1-6023-4194-b569-3a93d9f88447\") " pod="opendatahub/test-curl-7829-1623ae" Sep 08 17:09:30.162832 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:30.162749 2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7829-a87dbc" container="test-curl-7829-a87dbc" Sep 08 17:09:30.189734 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:30.189698 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7829-1623ae" Sep 08 17:09:30.469938 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:30.469910 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7829-1623ae"] Sep 08 17:09:30.471771 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:09:30.471741 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1ead5a1_6023_4194_b569_3a93d9f88447.slice/crio-2c9df3e7bc4eb85cf4f572a6d89d6877bc907f899ffbf98774a889e5be9d1700 WatchSource:0}: Error finding container 2c9df3e7bc4eb85cf4f572a6d89d6877bc907f899ffbf98774a889e5be9d1700: Status 404 returned error can't find the container with id 2c9df3e7bc4eb85cf4f572a6d89d6877bc907f899ffbf98774a889e5be9d1700 Sep 08 17:09:31.170918 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:31.170871 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7829-1623ae" event={"ID":"d1ead5a1-6023-4194-b569-3a93d9f88447","Type":"ContainerStarted","Data":"47b39d594b94222910b464abc37b73cb1d321fb0c42d52ae321fa88bba1cb065"} Sep 08 17:09:31.171363 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:31.170929 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7829-1623ae" event={"ID":"d1ead5a1-6023-4194-b569-3a93d9f88447","Type":"ContainerStarted","Data":"2c9df3e7bc4eb85cf4f572a6d89d6877bc907f899ffbf98774a889e5be9d1700"} Sep 08 17:09:31.171363 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:31.170923 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7829-a87dbc" podUID="3bfd2a2b-d4d6-4456-a4f1-9173b2df4740" containerName="test-curl-7829-a87dbc" containerID="cri-o://e472b2ac19d536fac821204e364f71e337aecd2d936a2ad09a16985ff9643722" gracePeriod=2 Sep 08 17:09:31.186652 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:31.186613 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7829-a87dbc\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="3bfd2a2b-d4d6-4456-a4f1-9173b2df4740" pod="opendatahub/test-curl-7829-a87dbc" Sep 08 17:09:31.187599 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:31.187560 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7829-1623ae" podStartSLOduration=2.187545723 podStartE2EDuration="2.187545723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:09:31.184895394 +0000 UTC m=+1133.921726510" watchObservedRunningTime="2026-09-08 17:09:31.187545723 +0000 UTC m=+1133.924376839" Sep 08 17:09:31.188511 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:31.188487 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7829-a87dbc\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="3bfd2a2b-d4d6-4456-a4f1-9173b2df4740" pod="opendatahub/test-curl-7829-a87dbc" Sep 08 17:09:31.788889 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:31.788855 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7829-1623ae"] Sep 08 17:09:31.796895 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:31.796360 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7829-1623ae"] Sep 08 17:09:31.917753 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:31.917673 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7829-0b2899"] Sep 08 17:09:31.918100 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:31.918087 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1ead5a1-6023-4194-b569-3a93d9f88447" containerName="test-curl-7829-1623ae" Sep 08 17:09:31.918144 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:31.918102 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1ead5a1-6023-4194-b569-3a93d9f88447" containerName="test-curl-7829-1623ae" Sep 08 17:09:31.918199 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:31.918190 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1ead5a1-6023-4194-b569-3a93d9f88447" containerName="test-curl-7829-1623ae" Sep 08 17:09:31.921402 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:31.921386 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7829-0b2899" Sep 08 17:09:31.929822 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:31.929797 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7829-0b2899"] Sep 08 17:09:32.089194 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:32.089151 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js2kn\" (UniqueName: \"kubernetes.io/projected/b776b1d8-f789-4f25-b454-e23254722d8e-kube-api-access-js2kn\") pod \"test-curl-7829-0b2899\" (UID: \"b776b1d8-f789-4f25-b454-e23254722d8e\") " pod="opendatahub/test-curl-7829-0b2899" Sep 08 17:09:32.177062 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:32.177032 2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7829-1623ae" container="test-curl-7829-1623ae" Sep 08 17:09:32.189599 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:32.189577 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js2kn\" (UniqueName: \"kubernetes.io/projected/b776b1d8-f789-4f25-b454-e23254722d8e-kube-api-access-js2kn\") pod \"test-curl-7829-0b2899\" (UID: \"b776b1d8-f789-4f25-b454-e23254722d8e\") " pod="opendatahub/test-curl-7829-0b2899" Sep 08 17:09:32.197632 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:32.197605 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js2kn\" (UniqueName: \"kubernetes.io/projected/b776b1d8-f789-4f25-b454-e23254722d8e-kube-api-access-js2kn\") pod \"test-curl-7829-0b2899\" (UID: \"b776b1d8-f789-4f25-b454-e23254722d8e\") " pod="opendatahub/test-curl-7829-0b2899" Sep 08 17:09:32.232102 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:32.232077 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7829-0b2899" Sep 08 17:09:32.515873 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:32.515843 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7829-0b2899"] Sep 08 17:09:32.517698 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:09:32.517665 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb776b1d8_f789_4f25_b454_e23254722d8e.slice/crio-cfebf0c2f77ada7410b383c3a249aa7bb984103a888c8417e201ac598fe7f2a3 WatchSource:0}: Error finding container cfebf0c2f77ada7410b383c3a249aa7bb984103a888c8417e201ac598fe7f2a3: Status 404 returned error can't find the container with id cfebf0c2f77ada7410b383c3a249aa7bb984103a888c8417e201ac598fe7f2a3 Sep 08 17:09:33.185030 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:33.184996 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7829-0b2899" event={"ID":"b776b1d8-f789-4f25-b454-e23254722d8e","Type":"ContainerStarted","Data":"0494690879abdbee0f36f7d95b10ff66b27b0d1c8400f3377b8748fb10b303f7"} Sep 08 17:09:33.185493 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:33.185040 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7829-0b2899" event={"ID":"b776b1d8-f789-4f25-b454-e23254722d8e","Type":"ContainerStarted","Data":"cfebf0c2f77ada7410b383c3a249aa7bb984103a888c8417e201ac598fe7f2a3"} Sep 08 17:09:33.185493 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:33.185137 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7829-1623ae" podUID="d1ead5a1-6023-4194-b569-3a93d9f88447" containerName="test-curl-7829-1623ae" containerID="cri-o://47b39d594b94222910b464abc37b73cb1d321fb0c42d52ae321fa88bba1cb065" gracePeriod=2 Sep 08 17:09:33.199777 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:33.199730 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7829-0b2899" podStartSLOduration=2.199713521 podStartE2EDuration="2.199713521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:09:33.19821388 +0000 UTC m=+1135.935044997" watchObservedRunningTime="2026-09-08 17:09:33.199713521 +0000 UTC m=+1135.936544636" Sep 08 17:09:33.199948 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:33.199921 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7829-1623ae\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="d1ead5a1-6023-4194-b569-3a93d9f88447" pod="opendatahub/test-curl-7829-1623ae" Sep 08 17:09:33.318740 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:33.318709 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7829-a87dbc" Sep 08 17:09:33.324150 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:33.324120 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7829-1623ae\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="d1ead5a1-6023-4194-b569-3a93d9f88447" pod="opendatahub/test-curl-7829-1623ae" Sep 08 17:09:33.402800 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:33.402759 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqsvp\" (UniqueName: \"kubernetes.io/projected/3bfd2a2b-d4d6-4456-a4f1-9173b2df4740-kube-api-access-lqsvp\") pod \"3bfd2a2b-d4d6-4456-a4f1-9173b2df4740\" (UID: \"3bfd2a2b-d4d6-4456-a4f1-9173b2df4740\") " Sep 08 17:09:33.404852 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:33.404818 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bfd2a2b-d4d6-4456-a4f1-9173b2df4740-kube-api-access-lqsvp" (OuterVolumeSpecName: "kube-api-access-lqsvp") pod "3bfd2a2b-d4d6-4456-a4f1-9173b2df4740" (UID: "3bfd2a2b-d4d6-4456-a4f1-9173b2df4740"). InnerVolumeSpecName "kube-api-access-lqsvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:09:33.503624 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:33.503552 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqsvp\" (UniqueName: \"kubernetes.io/projected/3bfd2a2b-d4d6-4456-a4f1-9173b2df4740-kube-api-access-lqsvp\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:09:33.761149 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:33.761074 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bfd2a2b-d4d6-4456-a4f1-9173b2df4740" path="/var/lib/kubelet/pods/3bfd2a2b-d4d6-4456-a4f1-9173b2df4740/volumes" Sep 08 17:09:33.866020 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:33.865982 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7829-0b2899"] Sep 08 17:09:33.871946 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:33.871914 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7829-0b2899"] Sep 08 17:09:33.989475 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:33.989442 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7829-119522"] Sep 08 17:09:33.989868 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:33.989856 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b776b1d8-f789-4f25-b454-e23254722d8e" containerName="test-curl-7829-0b2899" Sep 08 17:09:33.989914 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:33.989869 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b776b1d8-f789-4f25-b454-e23254722d8e" containerName="test-curl-7829-0b2899" Sep 08 17:09:33.989955 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:33.989942 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b776b1d8-f789-4f25-b454-e23254722d8e" containerName="test-curl-7829-0b2899" Sep 08 17:09:33.992748 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:33.992733 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7829-119522" Sep 08 17:09:33.998522 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:33.998493 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7829-119522"] Sep 08 17:09:34.108754 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:34.108662 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4brgp\" (UniqueName: \"kubernetes.io/projected/364d292c-001b-4afb-9c92-30d5fbce9a20-kube-api-access-4brgp\") pod \"test-curl-7829-119522\" (UID: \"364d292c-001b-4afb-9c92-30d5fbce9a20\") " pod="opendatahub/test-curl-7829-119522" Sep 08 17:09:34.191967 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:34.191935 2565 generic.go:356] "Generic (PLEG): container finished" podID="3bfd2a2b-d4d6-4456-a4f1-9173b2df4740" containerID="e472b2ac19d536fac821204e364f71e337aecd2d936a2ad09a16985ff9643722" exitCode=137 Sep 08 17:09:34.192349 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:34.191979 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7829-a87dbc" Sep 08 17:09:34.192349 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:34.192038 2565 scope.go:117] "RemoveContainer" containerID="e472b2ac19d536fac821204e364f71e337aecd2d936a2ad09a16985ff9643722" Sep 08 17:09:34.192349 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:34.192305 2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7829-0b2899" container="test-curl-7829-0b2899" Sep 08 17:09:34.201046 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:34.201023 2565 scope.go:117] "RemoveContainer" containerID="e472b2ac19d536fac821204e364f71e337aecd2d936a2ad09a16985ff9643722" Sep 08 17:09:34.201302 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:34.201282 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e472b2ac19d536fac821204e364f71e337aecd2d936a2ad09a16985ff9643722\": container with ID starting with e472b2ac19d536fac821204e364f71e337aecd2d936a2ad09a16985ff9643722 not found: ID does not exist" containerID="e472b2ac19d536fac821204e364f71e337aecd2d936a2ad09a16985ff9643722" Sep 08 17:09:34.201361 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:34.201311 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e472b2ac19d536fac821204e364f71e337aecd2d936a2ad09a16985ff9643722"} err="failed to get container status \"e472b2ac19d536fac821204e364f71e337aecd2d936a2ad09a16985ff9643722\": rpc error: code = NotFound desc = could not find container \"e472b2ac19d536fac821204e364f71e337aecd2d936a2ad09a16985ff9643722\": container with ID starting with e472b2ac19d536fac821204e364f71e337aecd2d936a2ad09a16985ff9643722 not found: ID does not exist" Sep 08 17:09:34.209716 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:34.209696 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4brgp\" (UniqueName: \"kubernetes.io/projected/364d292c-001b-4afb-9c92-30d5fbce9a20-kube-api-access-4brgp\") pod \"test-curl-7829-119522\" (UID: \"364d292c-001b-4afb-9c92-30d5fbce9a20\") " pod="opendatahub/test-curl-7829-119522" Sep 08 17:09:34.218861 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:34.218833 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4brgp\" (UniqueName: \"kubernetes.io/projected/364d292c-001b-4afb-9c92-30d5fbce9a20-kube-api-access-4brgp\") pod \"test-curl-7829-119522\" (UID: \"364d292c-001b-4afb-9c92-30d5fbce9a20\") " pod="opendatahub/test-curl-7829-119522" Sep 08 17:09:34.304854 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:34.304816 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7829-119522" Sep 08 17:09:34.584138 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:34.584108 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7829-119522"] Sep 08 17:09:34.584876 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:09:34.584848 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod364d292c_001b_4afb_9c92_30d5fbce9a20.slice/crio-3e37b656c38aaa4617db1c12f7fb3cd9c063931afa319b88edc4fe8ce51d0351 WatchSource:0}: Error finding container 3e37b656c38aaa4617db1c12f7fb3cd9c063931afa319b88edc4fe8ce51d0351: Status 404 returned error can't find the container with id 3e37b656c38aaa4617db1c12f7fb3cd9c063931afa319b88edc4fe8ce51d0351 Sep 08 17:09:35.199774 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:35.199739 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7829-119522" event={"ID":"364d292c-001b-4afb-9c92-30d5fbce9a20","Type":"ContainerStarted","Data":"0ce99a9c5f427a60766ef80e0239ecdcff6f1682ea5b9e797724bd3d71608829"} Sep 08 17:09:35.200097 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:35.199784 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7829-119522" event={"ID":"364d292c-001b-4afb-9c92-30d5fbce9a20","Type":"ContainerStarted","Data":"3e37b656c38aaa4617db1c12f7fb3cd9c063931afa319b88edc4fe8ce51d0351"} Sep 08 17:09:35.200900 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:35.200855 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7829-0b2899" podUID="b776b1d8-f789-4f25-b454-e23254722d8e" containerName="test-curl-7829-0b2899" containerID="cri-o://0494690879abdbee0f36f7d95b10ff66b27b0d1c8400f3377b8748fb10b303f7" gracePeriod=2 Sep 08 17:09:35.223963 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:35.223910 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7829-119522" podStartSLOduration=2.223895653 podStartE2EDuration="2.223895653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:09:35.221715056 +0000 UTC m=+1137.958546174" watchObservedRunningTime="2026-09-08 17:09:35.223895653 +0000 UTC m=+1137.960726769" Sep 08 17:09:35.224559 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:35.224490 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7829-0b2899\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="b776b1d8-f789-4f25-b454-e23254722d8e" pod="opendatahub/test-curl-7829-0b2899" Sep 08 17:09:35.370040 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:35.370011 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7829-1623ae" Sep 08 17:09:35.372680 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:35.372651 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7829-0b2899\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="b776b1d8-f789-4f25-b454-e23254722d8e" pod="opendatahub/test-curl-7829-0b2899" Sep 08 17:09:35.521028 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:35.520944 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h45jf\" (UniqueName: \"kubernetes.io/projected/d1ead5a1-6023-4194-b569-3a93d9f88447-kube-api-access-h45jf\") pod \"d1ead5a1-6023-4194-b569-3a93d9f88447\" (UID: \"d1ead5a1-6023-4194-b569-3a93d9f88447\") " Sep 08 17:09:35.522743 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:35.522721 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ead5a1-6023-4194-b569-3a93d9f88447-kube-api-access-h45jf" (OuterVolumeSpecName: "kube-api-access-h45jf") pod "d1ead5a1-6023-4194-b569-3a93d9f88447" (UID: "d1ead5a1-6023-4194-b569-3a93d9f88447"). InnerVolumeSpecName "kube-api-access-h45jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:09:35.621947 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:35.621915 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h45jf\" (UniqueName: \"kubernetes.io/projected/d1ead5a1-6023-4194-b569-3a93d9f88447-kube-api-access-h45jf\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:09:35.760796 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:35.760756 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1ead5a1-6023-4194-b569-3a93d9f88447" path="/var/lib/kubelet/pods/d1ead5a1-6023-4194-b569-3a93d9f88447/volumes" Sep 08 17:09:35.969647 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:35.969610 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7829-119522"] Sep 08 17:09:35.976724 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:35.976693 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7829-119522"] Sep 08 17:09:36.207743 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:36.207710 2565 generic.go:356] "Generic (PLEG): container finished" podID="d1ead5a1-6023-4194-b569-3a93d9f88447" containerID="47b39d594b94222910b464abc37b73cb1d321fb0c42d52ae321fa88bba1cb065" exitCode=137 Sep 08 17:09:36.208201 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:36.207750 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7829-1623ae" Sep 08 17:09:36.208201 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:36.207808 2565 scope.go:117] "RemoveContainer" containerID="47b39d594b94222910b464abc37b73cb1d321fb0c42d52ae321fa88bba1cb065" Sep 08 17:09:36.208201 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:36.208099 2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7829-119522" container="test-curl-7829-119522" Sep 08 17:09:36.216770 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:36.216750 2565 scope.go:117] "RemoveContainer" containerID="47b39d594b94222910b464abc37b73cb1d321fb0c42d52ae321fa88bba1cb065" Sep 08 17:09:36.217039 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:36.217020 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47b39d594b94222910b464abc37b73cb1d321fb0c42d52ae321fa88bba1cb065\": container with ID starting with 47b39d594b94222910b464abc37b73cb1d321fb0c42d52ae321fa88bba1cb065 not found: ID does not exist" containerID="47b39d594b94222910b464abc37b73cb1d321fb0c42d52ae321fa88bba1cb065" Sep 08 17:09:36.217103 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:36.217047 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47b39d594b94222910b464abc37b73cb1d321fb0c42d52ae321fa88bba1cb065"} err="failed to get container status \"47b39d594b94222910b464abc37b73cb1d321fb0c42d52ae321fa88bba1cb065\": rpc error: code = NotFound desc = could not find container \"47b39d594b94222910b464abc37b73cb1d321fb0c42d52ae321fa88bba1cb065\": container with ID starting with 47b39d594b94222910b464abc37b73cb1d321fb0c42d52ae321fa88bba1cb065 not found: ID does not exist" Sep 08 17:09:36.611269 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:36.611236 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:37.128321 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:37.128288 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:37.214913 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:37.214878 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7829-119522" podUID="364d292c-001b-4afb-9c92-30d5fbce9a20" containerName="test-curl-7829-119522" containerID="cri-o://0ce99a9c5f427a60766ef80e0239ecdcff6f1682ea5b9e797724bd3d71608829" gracePeriod=2 Sep 08 17:09:37.218607 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:37.218579 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7829-119522\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="364d292c-001b-4afb-9c92-30d5fbce9a20" pod="opendatahub/test-curl-7829-119522" Sep 08 17:09:37.376484 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:37.376457 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7829-0b2899" Sep 08 17:09:37.378946 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:37.378886 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7829-119522\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="364d292c-001b-4afb-9c92-30d5fbce9a20" pod="opendatahub/test-curl-7829-119522" Sep 08 17:09:37.396069 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:37.396037 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:37.540042 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:37.540010 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-js2kn\" (UniqueName: \"kubernetes.io/projected/b776b1d8-f789-4f25-b454-e23254722d8e-kube-api-access-js2kn\") pod \"b776b1d8-f789-4f25-b454-e23254722d8e\" (UID: \"b776b1d8-f789-4f25-b454-e23254722d8e\") " Sep 08 17:09:37.542082 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:37.542053 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b776b1d8-f789-4f25-b454-e23254722d8e-kube-api-access-js2kn" (OuterVolumeSpecName: "kube-api-access-js2kn") pod "b776b1d8-f789-4f25-b454-e23254722d8e" (UID: "b776b1d8-f789-4f25-b454-e23254722d8e"). InnerVolumeSpecName "kube-api-access-js2kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:09:37.641679 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:37.641596 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js2kn\" (UniqueName: \"kubernetes.io/projected/b776b1d8-f789-4f25-b454-e23254722d8e-kube-api-access-js2kn\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:09:37.760282 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:37.760243 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7829-119522\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="364d292c-001b-4afb-9c92-30d5fbce9a20" pod="opendatahub/test-curl-7829-119522" Sep 08 17:09:37.761950 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:37.761926 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b776b1d8-f789-4f25-b454-e23254722d8e" path="/var/lib/kubelet/pods/b776b1d8-f789-4f25-b454-e23254722d8e/volumes" Sep 08 17:09:38.222246 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:38.222204 2565 generic.go:356] "Generic (PLEG): container finished" podID="b776b1d8-f789-4f25-b454-e23254722d8e" containerID="0494690879abdbee0f36f7d95b10ff66b27b0d1c8400f3377b8748fb10b303f7" exitCode=137 Sep 08 17:09:38.222246 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:38.222251 2565 scope.go:117] "RemoveContainer" containerID="0494690879abdbee0f36f7d95b10ff66b27b0d1c8400f3377b8748fb10b303f7" Sep 08 17:09:38.222772 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:38.222252 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7829-0b2899" Sep 08 17:09:38.232348 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:38.232328 2565 scope.go:117] "RemoveContainer" containerID="0494690879abdbee0f36f7d95b10ff66b27b0d1c8400f3377b8748fb10b303f7" Sep 08 17:09:38.232647 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:38.232625 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0494690879abdbee0f36f7d95b10ff66b27b0d1c8400f3377b8748fb10b303f7\": container with ID starting with 0494690879abdbee0f36f7d95b10ff66b27b0d1c8400f3377b8748fb10b303f7 not found: ID does not exist" containerID="0494690879abdbee0f36f7d95b10ff66b27b0d1c8400f3377b8748fb10b303f7" Sep 08 17:09:38.232736 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:38.232655 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0494690879abdbee0f36f7d95b10ff66b27b0d1c8400f3377b8748fb10b303f7"} err="failed to get container status \"0494690879abdbee0f36f7d95b10ff66b27b0d1c8400f3377b8748fb10b303f7\": rpc error: code = NotFound desc = could not find container \"0494690879abdbee0f36f7d95b10ff66b27b0d1c8400f3377b8748fb10b303f7\": container with ID starting with 0494690879abdbee0f36f7d95b10ff66b27b0d1c8400f3377b8748fb10b303f7 not found: ID does not exist" Sep 08 17:09:39.409898 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:39.409872 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7829-119522" Sep 08 17:09:39.559014 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:39.558926 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4brgp\" (UniqueName: \"kubernetes.io/projected/364d292c-001b-4afb-9c92-30d5fbce9a20-kube-api-access-4brgp\") pod \"364d292c-001b-4afb-9c92-30d5fbce9a20\" (UID: \"364d292c-001b-4afb-9c92-30d5fbce9a20\") " Sep 08 17:09:39.560931 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:39.560910 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/364d292c-001b-4afb-9c92-30d5fbce9a20-kube-api-access-4brgp" (OuterVolumeSpecName: "kube-api-access-4brgp") pod "364d292c-001b-4afb-9c92-30d5fbce9a20" (UID: "364d292c-001b-4afb-9c92-30d5fbce9a20"). InnerVolumeSpecName "kube-api-access-4brgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:09:39.660296 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:39.660256 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4brgp\" (UniqueName: \"kubernetes.io/projected/364d292c-001b-4afb-9c92-30d5fbce9a20-kube-api-access-4brgp\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:09:39.761368 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:39.761335 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="364d292c-001b-4afb-9c92-30d5fbce9a20" path="/var/lib/kubelet/pods/364d292c-001b-4afb-9c92-30d5fbce9a20/volumes" Sep 08 17:09:40.235865 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:40.235831 2565 generic.go:356] "Generic (PLEG): container finished" podID="364d292c-001b-4afb-9c92-30d5fbce9a20" containerID="0ce99a9c5f427a60766ef80e0239ecdcff6f1682ea5b9e797724bd3d71608829" exitCode=137 Sep 08 17:09:40.236047 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:40.235874 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7829-119522" Sep 08 17:09:40.236047 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:40.235908 2565 scope.go:117] "RemoveContainer" containerID="0ce99a9c5f427a60766ef80e0239ecdcff6f1682ea5b9e797724bd3d71608829" Sep 08 17:09:40.244868 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:40.244844 2565 scope.go:117] "RemoveContainer" containerID="0ce99a9c5f427a60766ef80e0239ecdcff6f1682ea5b9e797724bd3d71608829" Sep 08 17:09:40.245139 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:40.245119 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ce99a9c5f427a60766ef80e0239ecdcff6f1682ea5b9e797724bd3d71608829\": container with ID starting with 0ce99a9c5f427a60766ef80e0239ecdcff6f1682ea5b9e797724bd3d71608829 not found: ID does not exist" containerID="0ce99a9c5f427a60766ef80e0239ecdcff6f1682ea5b9e797724bd3d71608829" Sep 08 17:09:40.245228 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:40.245147 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ce99a9c5f427a60766ef80e0239ecdcff6f1682ea5b9e797724bd3d71608829"} err="failed to get container status \"0ce99a9c5f427a60766ef80e0239ecdcff6f1682ea5b9e797724bd3d71608829\": rpc error: code = NotFound desc = could not find container \"0ce99a9c5f427a60766ef80e0239ecdcff6f1682ea5b9e797724bd3d71608829\": container with ID starting with 0ce99a9c5f427a60766ef80e0239ecdcff6f1682ea5b9e797724bd3d71608829 not found: ID does not exist" Sep 08 17:09:40.414868 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:40.414787 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:40.552963 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:40.552929 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:41.333364 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:41.333331 2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb776b1d8_f789_4f25_b454_e23254722d8e.slice/crio-cfebf0c2f77ada7410b383c3a249aa7bb984103a888c8417e201ac598fe7f2a3\": RecentStats: unable to find data in memory cache]" Sep 08 17:09:41.577108 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.577075 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs"] Sep 08 17:09:41.577613 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.577594 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="364d292c-001b-4afb-9c92-30d5fbce9a20" containerName="test-curl-7829-119522" Sep 08 17:09:41.577666 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.577616 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="364d292c-001b-4afb-9c92-30d5fbce9a20" containerName="test-curl-7829-119522" Sep 08 17:09:41.577721 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.577701 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="364d292c-001b-4afb-9c92-30d5fbce9a20" containerName="test-curl-7829-119522" Sep 08 17:09:41.582288 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.582268 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" Sep 08 17:09:41.585660 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.585608 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-cb1565be-serving-cert\"" Sep 08 17:09:41.594465 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.594441 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs"] Sep 08 17:09:41.620271 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.620248 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:41.678488 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.678454 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsp9f\" (UniqueName: \"kubernetes.io/projected/57351169-6a46-45e0-8ea5-46930f947d15-kube-api-access-nsp9f\") pod \"maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs\" (UID: \"57351169-6a46-45e0-8ea5-46930f947d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" Sep 08 17:09:41.678632 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.678495 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57351169-6a46-45e0-8ea5-46930f947d15-maas-api-tls\") pod \"maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs\" (UID: \"57351169-6a46-45e0-8ea5-46930f947d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" Sep 08 17:09:41.678632 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.678533 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57351169-6a46-45e0-8ea5-46930f947d15-metrics-certs\") pod \"maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs\" (UID: \"57351169-6a46-45e0-8ea5-46930f947d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" Sep 08 17:09:41.678731 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.678712 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57351169-6a46-45e0-8ea5-46930f947d15-openshift-service-ca\") pod \"maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs\" (UID: \"57351169-6a46-45e0-8ea5-46930f947d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" Sep 08 17:09:41.749773 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.749738 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:41.779802 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.779773 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57351169-6a46-45e0-8ea5-46930f947d15-openshift-service-ca\") pod \"maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs\" (UID: \"57351169-6a46-45e0-8ea5-46930f947d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" Sep 08 17:09:41.779979 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.779811 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsp9f\" (UniqueName: \"kubernetes.io/projected/57351169-6a46-45e0-8ea5-46930f947d15-kube-api-access-nsp9f\") pod \"maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs\" (UID: \"57351169-6a46-45e0-8ea5-46930f947d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" Sep 08 17:09:41.779979 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.779830 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57351169-6a46-45e0-8ea5-46930f947d15-maas-api-tls\") pod \"maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs\" (UID: \"57351169-6a46-45e0-8ea5-46930f947d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" Sep 08 17:09:41.779979 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.779845 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57351169-6a46-45e0-8ea5-46930f947d15-metrics-certs\") pod \"maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs\" (UID: \"57351169-6a46-45e0-8ea5-46930f947d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" Sep 08 17:09:41.780438 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.780394 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57351169-6a46-45e0-8ea5-46930f947d15-openshift-service-ca\") pod \"maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs\" (UID: \"57351169-6a46-45e0-8ea5-46930f947d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" Sep 08 17:09:41.782387 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.782366 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57351169-6a46-45e0-8ea5-46930f947d15-metrics-certs\") pod \"maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs\" (UID: \"57351169-6a46-45e0-8ea5-46930f947d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" Sep 08 17:09:41.782517 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.782393 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57351169-6a46-45e0-8ea5-46930f947d15-maas-api-tls\") pod \"maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs\" (UID: \"57351169-6a46-45e0-8ea5-46930f947d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" Sep 08 17:09:41.791046 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.791026 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsp9f\" (UniqueName: \"kubernetes.io/projected/57351169-6a46-45e0-8ea5-46930f947d15-kube-api-access-nsp9f\") pod \"maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs\" (UID: \"57351169-6a46-45e0-8ea5-46930f947d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" Sep 08 17:09:41.872196 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.872119 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:41.892931 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:41.892903 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" Sep 08 17:09:42.184636 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:42.184603 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs"] Sep 08 17:09:42.185572 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:09:42.185546 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57351169_6a46_45e0_8ea5_46930f947d15.slice/crio-1dfca15a2c0b38da9bc5fa4dbf8e8ad5e2082de66be681f9901880923d1bdee1 WatchSource:0}: Error finding container 1dfca15a2c0b38da9bc5fa4dbf8e8ad5e2082de66be681f9901880923d1bdee1: Status 404 returned error can't find the container with id 1dfca15a2c0b38da9bc5fa4dbf8e8ad5e2082de66be681f9901880923d1bdee1 Sep 08 17:09:42.250621 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:42.250572 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" event={"ID":"57351169-6a46-45e0-8ea5-46930f947d15","Type":"ContainerStarted","Data":"1dfca15a2c0b38da9bc5fa4dbf8e8ad5e2082de66be681f9901880923d1bdee1"} Sep 08 17:09:42.359845 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:42.359809 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:42.566227 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:42.566134 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:44.265109 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:44.265070 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" event={"ID":"57351169-6a46-45e0-8ea5-46930f947d15","Type":"ContainerStarted","Data":"143bd62fc4b23ddc72f9b2690307a240adfb7cc210c531b0560565b374b20bef"} Sep 08 17:09:44.265516 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:44.265260 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" Sep 08 17:09:44.285581 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:44.285534 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" podStartSLOduration=1.915159873 podStartE2EDuration="3.285516764s" totalImagesPullingTime="1.370356891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:09:44.28229169 +0000 UTC m=+1147.019122817" watchObservedRunningTime="2026-09-08 17:09:44.285516764 +0000 UTC m=+1147.022347881" Sep 08 17:09:50.276051 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:50.276019 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" Sep 08 17:09:51.390298 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:51.390266 2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb776b1d8_f789_4f25_b454_e23254722d8e.slice/crio-cfebf0c2f77ada7410b383c3a249aa7bb984103a888c8417e201ac598fe7f2a3\": RecentStats: unable to find data in memory cache]" Sep 08 17:09:53.823300 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:53.823262 2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb776b1d8_f789_4f25_b454_e23254722d8e.slice/crio-cfebf0c2f77ada7410b383c3a249aa7bb984103a888c8417e201ac598fe7f2a3\": RecentStats: unable to find data in memory cache]" Sep 08 17:09:58.254017 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:09:58.253976 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:09:59.994894 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:59.994854 2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb776b1d8_f789_4f25_b454_e23254722d8e.slice/crio-cfebf0c2f77ada7410b383c3a249aa7bb984103a888c8417e201ac598fe7f2a3\": RecentStats: unable to find data in memory cache]" Sep 08 17:09:59.995567 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:09:59.995538 2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb776b1d8_f789_4f25_b454_e23254722d8e.slice/crio-cfebf0c2f77ada7410b383c3a249aa7bb984103a888c8417e201ac598fe7f2a3\": RecentStats: unable to find data in memory cache]" Sep 08 17:10:01.401652 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:10:01.401616 2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb776b1d8_f789_4f25_b454_e23254722d8e.slice/crio-cfebf0c2f77ada7410b383c3a249aa7bb984103a888c8417e201ac598fe7f2a3\": RecentStats: unable to find data in memory cache]" Sep 08 17:10:01.573885 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:10:01.573836 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:10:04.077887 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:10:04.077844 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:10:05.144441 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:10:05.144394 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:10:08.875159 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:10:08.875125 2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb776b1d8_f789_4f25_b454_e23254722d8e.slice/crio-cfebf0c2f77ada7410b383c3a249aa7bb984103a888c8417e201ac598fe7f2a3\": RecentStats: unable to find data in memory cache]" Sep 08 17:10:11.411734 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:10:11.411695 2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb776b1d8_f789_4f25_b454_e23254722d8e.slice/crio-cfebf0c2f77ada7410b383c3a249aa7bb984103a888c8417e201ac598fe7f2a3\": RecentStats: unable to find data in memory cache]" Sep 08 17:10:21.465857 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:10:21.465823 2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb776b1d8_f789_4f25_b454_e23254722d8e.slice/crio-cfebf0c2f77ada7410b383c3a249aa7bb984103a888c8417e201ac598fe7f2a3\": RecentStats: unable to find data in memory cache]" Sep 08 17:10:23.336671 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:10:23.336636 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:10:23.821224 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:10:23.821184 2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb776b1d8_f789_4f25_b454_e23254722d8e.slice/crio-cfebf0c2f77ada7410b383c3a249aa7bb984103a888c8417e201ac598fe7f2a3\": RecentStats: unable to find data in memory cache]" Sep 08 17:10:31.522661 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:10:31.522622 2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb776b1d8_f789_4f25_b454_e23254722d8e.slice/crio-cfebf0c2f77ada7410b383c3a249aa7bb984103a888c8417e201ac598fe7f2a3\": RecentStats: unable to find data in memory cache]" Sep 08 17:10:37.823254 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:10:37.823220 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/2.log" Sep 08 17:10:37.826230 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:10:37.826197 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/2.log" Sep 08 17:10:37.831107 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:10:37.831083 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xxnl_bf53ac63-f864-402b-bf12-45b289e7e13d/ovn-acl-logging/0.log" Sep 08 17:10:37.834104 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:10:37.834087 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xxnl_bf53ac63-f864-402b-bf12-45b289e7e13d/ovn-acl-logging/0.log" Sep 08 17:10:51.755688 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:10:51.755652 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:10:51.810779 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:10:51.810740 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:11:05.407172 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:05.407127 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs"] Sep 08 17:11:05.443879 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:05.443847 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:11:05.543885 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:05.543848 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:11:05.645691 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:05.645651 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:11:05.708552 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:05.708522 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:11:05.752220 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:05.752171 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" podUID="57351169-6a46-45e0-8ea5-46930f947d15" containerName="maas-api" containerID="cri-o://143bd62fc4b23ddc72f9b2690307a240adfb7cc210c531b0560565b374b20bef" gracePeriod=30 Sep 08 17:11:06.005187 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:06.005162 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" Sep 08 17:11:06.170451 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:06.170399 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsp9f\" (UniqueName: \"kubernetes.io/projected/57351169-6a46-45e0-8ea5-46930f947d15-kube-api-access-nsp9f\") pod \"57351169-6a46-45e0-8ea5-46930f947d15\" (UID: \"57351169-6a46-45e0-8ea5-46930f947d15\") " Sep 08 17:11:06.170613 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:06.170461 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57351169-6a46-45e0-8ea5-46930f947d15-metrics-certs\") pod \"57351169-6a46-45e0-8ea5-46930f947d15\" (UID: \"57351169-6a46-45e0-8ea5-46930f947d15\") " Sep 08 17:11:06.170613 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:06.170503 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57351169-6a46-45e0-8ea5-46930f947d15-maas-api-tls\") pod \"57351169-6a46-45e0-8ea5-46930f947d15\" (UID: \"57351169-6a46-45e0-8ea5-46930f947d15\") " Sep 08 17:11:06.170613 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:06.170539 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57351169-6a46-45e0-8ea5-46930f947d15-openshift-service-ca\") pod \"57351169-6a46-45e0-8ea5-46930f947d15\" (UID: \"57351169-6a46-45e0-8ea5-46930f947d15\") " Sep 08 17:11:06.170970 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:06.170936 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57351169-6a46-45e0-8ea5-46930f947d15-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "57351169-6a46-45e0-8ea5-46930f947d15" (UID: "57351169-6a46-45e0-8ea5-46930f947d15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:11:06.172622 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:06.172588 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57351169-6a46-45e0-8ea5-46930f947d15-kube-api-access-nsp9f" (OuterVolumeSpecName: "kube-api-access-nsp9f") pod "57351169-6a46-45e0-8ea5-46930f947d15" (UID: "57351169-6a46-45e0-8ea5-46930f947d15"). InnerVolumeSpecName "kube-api-access-nsp9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:11:06.172750 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:06.172594 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57351169-6a46-45e0-8ea5-46930f947d15-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "57351169-6a46-45e0-8ea5-46930f947d15" (UID: "57351169-6a46-45e0-8ea5-46930f947d15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:11:06.172750 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:06.172664 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57351169-6a46-45e0-8ea5-46930f947d15-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "57351169-6a46-45e0-8ea5-46930f947d15" (UID: "57351169-6a46-45e0-8ea5-46930f947d15"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:11:06.271410 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:06.271320 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsp9f\" (UniqueName: \"kubernetes.io/projected/57351169-6a46-45e0-8ea5-46930f947d15-kube-api-access-nsp9f\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:11:06.271410 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:06.271348 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57351169-6a46-45e0-8ea5-46930f947d15-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:11:06.271410 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:06.271358 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57351169-6a46-45e0-8ea5-46930f947d15-maas-api-tls\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:11:06.271410 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:06.271369 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57351169-6a46-45e0-8ea5-46930f947d15-openshift-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:11:06.759175 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:06.759141 2565 generic.go:356] "Generic (PLEG): container finished" podID="57351169-6a46-45e0-8ea5-46930f947d15" containerID="143bd62fc4b23ddc72f9b2690307a240adfb7cc210c531b0560565b374b20bef" exitCode=0 Sep 08 17:11:06.759664 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:06.759190 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" Sep 08 17:11:06.759664 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:06.759227 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" event={"ID":"57351169-6a46-45e0-8ea5-46930f947d15","Type":"ContainerDied","Data":"143bd62fc4b23ddc72f9b2690307a240adfb7cc210c531b0560565b374b20bef"} Sep 08 17:11:06.759664 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:06.759300 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs" event={"ID":"57351169-6a46-45e0-8ea5-46930f947d15","Type":"ContainerDied","Data":"1dfca15a2c0b38da9bc5fa4dbf8e8ad5e2082de66be681f9901880923d1bdee1"} Sep 08 17:11:06.759664 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:06.759338 2565 scope.go:117] "RemoveContainer" containerID="143bd62fc4b23ddc72f9b2690307a240adfb7cc210c531b0560565b374b20bef" Sep 08 17:11:06.769961 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:06.769941 2565 scope.go:117] "RemoveContainer" containerID="143bd62fc4b23ddc72f9b2690307a240adfb7cc210c531b0560565b374b20bef" Sep 08 17:11:06.770229 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:11:06.770207 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"143bd62fc4b23ddc72f9b2690307a240adfb7cc210c531b0560565b374b20bef\": container with ID starting with 143bd62fc4b23ddc72f9b2690307a240adfb7cc210c531b0560565b374b20bef not found: ID does not exist" containerID="143bd62fc4b23ddc72f9b2690307a240adfb7cc210c531b0560565b374b20bef" Sep 08 17:11:06.770278 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:06.770237 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"143bd62fc4b23ddc72f9b2690307a240adfb7cc210c531b0560565b374b20bef"} err="failed to get container status \"143bd62fc4b23ddc72f9b2690307a240adfb7cc210c531b0560565b374b20bef\": rpc error: code = NotFound desc = could not find container \"143bd62fc4b23ddc72f9b2690307a240adfb7cc210c531b0560565b374b20bef\": container with ID starting with 143bd62fc4b23ddc72f9b2690307a240adfb7cc210c531b0560565b374b20bef not found: ID does not exist" Sep 08 17:11:06.781573 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:06.781551 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs"] Sep 08 17:11:06.786536 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:06.786513 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb1565be-7fc798df66-hw2hs"] Sep 08 17:11:07.762264 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:07.762231 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57351169-6a46-45e0-8ea5-46930f947d15" path="/var/lib/kubelet/pods/57351169-6a46-45e0-8ea5-46930f947d15/volumes" Sep 08 17:11:08.670155 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:08.670126 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:11:13.743280 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:13.743239 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:11:50.261547 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:50.261517 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:11:51.335010 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:51.334973 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:11:56.293133 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:56.293090 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:11:56.398919 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:56.398881 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:11:56.539095 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:56.539059 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:11:57.759827 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:57.759792 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:11:57.859606 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:57.859570 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:11:57.959586 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:57.959547 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:11:58.528019 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:58.527969 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:11:58.627850 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:58.627815 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:11:58.727163 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:58.727128 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:11:59.433887 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:59.433853 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:11:59.534943 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:59.534905 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:11:59.947764 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:11:59.947726 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:05.788995 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:05.788955 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:05.862987 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:05.862946 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:06.416158 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:06.416125 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:12.346759 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:12.346713 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:12.525657 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:12.525623 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:12.625487 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:12.625375 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:22.993959 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:22.993921 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:23.205969 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:23.205931 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:30.902070 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:30.902034 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7832-6a99ee"] Sep 08 17:12:30.902483 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:30.902468 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57351169-6a46-45e0-8ea5-46930f947d15" containerName="maas-api" Sep 08 17:12:30.902483 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:30.902484 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="57351169-6a46-45e0-8ea5-46930f947d15" containerName="maas-api" Sep 08 17:12:30.902575 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:30.902564 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="57351169-6a46-45e0-8ea5-46930f947d15" containerName="maas-api" Sep 08 17:12:30.905787 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:30.905771 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-6a99ee" Sep 08 17:12:30.908311 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:30.908284 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v8zrb\"" Sep 08 17:12:30.910678 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:30.910660 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7832-6a99ee"] Sep 08 17:12:31.054196 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:31.054144 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rncqj\" (UniqueName: \"kubernetes.io/projected/319eda3b-896d-4a07-9541-4665dffc8c40-kube-api-access-rncqj\") pod \"test-curl-7832-6a99ee\" (UID: \"319eda3b-896d-4a07-9541-4665dffc8c40\") " pod="opendatahub/test-curl-7832-6a99ee" Sep 08 17:12:31.155799 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:31.155709 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rncqj\" (UniqueName: \"kubernetes.io/projected/319eda3b-896d-4a07-9541-4665dffc8c40-kube-api-access-rncqj\") pod \"test-curl-7832-6a99ee\" (UID: \"319eda3b-896d-4a07-9541-4665dffc8c40\") " pod="opendatahub/test-curl-7832-6a99ee" Sep 08 17:12:31.163850 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:31.163825 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rncqj\" (UniqueName: \"kubernetes.io/projected/319eda3b-896d-4a07-9541-4665dffc8c40-kube-api-access-rncqj\") pod \"test-curl-7832-6a99ee\" (UID: \"319eda3b-896d-4a07-9541-4665dffc8c40\") " pod="opendatahub/test-curl-7832-6a99ee" Sep 08 17:12:31.215502 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:31.215472 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-6a99ee" Sep 08 17:12:31.486435 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:31.486386 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7832-6a99ee"] Sep 08 17:12:31.487730 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:12:31.487706 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod319eda3b_896d_4a07_9541_4665dffc8c40.slice/crio-b3ea00d71d2f0426096848f3fe0ba19b3a7924001f2ccba3e0d29809e1be0542 WatchSource:0}: Error finding container b3ea00d71d2f0426096848f3fe0ba19b3a7924001f2ccba3e0d29809e1be0542: Status 404 returned error can't find the container with id b3ea00d71d2f0426096848f3fe0ba19b3a7924001f2ccba3e0d29809e1be0542 Sep 08 17:12:32.267758 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:32.267717 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7832-6a99ee" event={"ID":"319eda3b-896d-4a07-9541-4665dffc8c40","Type":"ContainerStarted","Data":"cf8c2caba45e0c97711c75bc8b4448936748fdddc4a46f689f056dfbb9af92b7"} Sep 08 17:12:32.268148 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:32.267768 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7832-6a99ee" event={"ID":"319eda3b-896d-4a07-9541-4665dffc8c40","Type":"ContainerStarted","Data":"b3ea00d71d2f0426096848f3fe0ba19b3a7924001f2ccba3e0d29809e1be0542"} Sep 08 17:12:32.283260 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:32.283213 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7832-6a99ee" podStartSLOduration=2.283196824 podStartE2EDuration="2.283196824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:12:32.280374082 +0000 UTC m=+1315.017205200" watchObservedRunningTime="2026-09-08 17:12:32.283196824 +0000 UTC m=+1315.020027940" Sep 08 17:12:32.851855 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:32.851810 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7832-6a99ee"] Sep 08 17:12:32.856877 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:32.856843 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7832-6a99ee"] Sep 08 17:12:33.273768 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:12:33.273736 2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7832-6a99ee" container="test-curl-7832-6a99ee" Sep 08 17:12:34.173029 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:34.172986 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:34.279151 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:34.279091 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7832-6a99ee" podUID="319eda3b-896d-4a07-9541-4665dffc8c40" containerName="test-curl-7832-6a99ee" containerID="cri-o://cf8c2caba45e0c97711c75bc8b4448936748fdddc4a46f689f056dfbb9af92b7" gracePeriod=2 Sep 08 17:12:35.300597 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:35.300570 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:36.471404 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:36.471373 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-6a99ee" Sep 08 17:12:36.606054 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:36.605946 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rncqj\" (UniqueName: \"kubernetes.io/projected/319eda3b-896d-4a07-9541-4665dffc8c40-kube-api-access-rncqj\") pod \"319eda3b-896d-4a07-9541-4665dffc8c40\" (UID: \"319eda3b-896d-4a07-9541-4665dffc8c40\") " Sep 08 17:12:36.608062 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:36.608030 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319eda3b-896d-4a07-9541-4665dffc8c40-kube-api-access-rncqj" (OuterVolumeSpecName: "kube-api-access-rncqj") pod "319eda3b-896d-4a07-9541-4665dffc8c40" (UID: "319eda3b-896d-4a07-9541-4665dffc8c40"). InnerVolumeSpecName "kube-api-access-rncqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:12:36.707743 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:36.707696 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rncqj\" (UniqueName: \"kubernetes.io/projected/319eda3b-896d-4a07-9541-4665dffc8c40-kube-api-access-rncqj\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:12:37.298271 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:37.298234 2565 generic.go:356] "Generic (PLEG): container finished" podID="319eda3b-896d-4a07-9541-4665dffc8c40" containerID="cf8c2caba45e0c97711c75bc8b4448936748fdddc4a46f689f056dfbb9af92b7" exitCode=137 Sep 08 17:12:37.298474 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:37.298274 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-6a99ee" Sep 08 17:12:37.298474 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:37.298287 2565 scope.go:117] "RemoveContainer" containerID="cf8c2caba45e0c97711c75bc8b4448936748fdddc4a46f689f056dfbb9af92b7" Sep 08 17:12:37.307503 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:37.307479 2565 scope.go:117] "RemoveContainer" containerID="cf8c2caba45e0c97711c75bc8b4448936748fdddc4a46f689f056dfbb9af92b7" Sep 08 17:12:37.307757 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:12:37.307738 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf8c2caba45e0c97711c75bc8b4448936748fdddc4a46f689f056dfbb9af92b7\": container with ID starting with cf8c2caba45e0c97711c75bc8b4448936748fdddc4a46f689f056dfbb9af92b7 not found: ID does not exist" containerID="cf8c2caba45e0c97711c75bc8b4448936748fdddc4a46f689f056dfbb9af92b7" Sep 08 17:12:37.307828 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:37.307770 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf8c2caba45e0c97711c75bc8b4448936748fdddc4a46f689f056dfbb9af92b7"} err="failed to get container status \"cf8c2caba45e0c97711c75bc8b4448936748fdddc4a46f689f056dfbb9af92b7\": rpc error: code = NotFound desc = could not find container \"cf8c2caba45e0c97711c75bc8b4448936748fdddc4a46f689f056dfbb9af92b7\": container with ID starting with cf8c2caba45e0c97711c75bc8b4448936748fdddc4a46f689f056dfbb9af92b7 not found: ID does not exist" Sep 08 17:12:37.308252 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:12:37.308234 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7832-6a99ee\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="319eda3b-896d-4a07-9541-4665dffc8c40" pod="opendatahub/test-curl-7832-6a99ee" Sep 08 17:12:37.758812 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:12:37.758777 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7832-6a99ee\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="319eda3b-896d-4a07-9541-4665dffc8c40" pod="opendatahub/test-curl-7832-6a99ee" Sep 08 17:12:37.760826 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:37.760803 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="319eda3b-896d-4a07-9541-4665dffc8c40" path="/var/lib/kubelet/pods/319eda3b-896d-4a07-9541-4665dffc8c40/volumes" Sep 08 17:12:44.108783 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:44.108748 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:49.073251 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:49.073210 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:49.174008 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:49.173971 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:49.917173 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:49.917132 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:49.982131 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:49.982099 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:50.268972 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:50.268938 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:50.661442 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:50.661335 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:50.934664 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:50.934633 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:53.321443 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:53.321391 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:53.420496 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:53.420460 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:54.519516 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.519469 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp"] Sep 08 17:12:54.520175 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.520150 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="319eda3b-896d-4a07-9541-4665dffc8c40" containerName="test-curl-7832-6a99ee" Sep 08 17:12:54.520247 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.520178 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="319eda3b-896d-4a07-9541-4665dffc8c40" containerName="test-curl-7832-6a99ee" Sep 08 17:12:54.520314 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.520301 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="319eda3b-896d-4a07-9541-4665dffc8c40" containerName="test-curl-7832-6a99ee" Sep 08 17:12:54.525390 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.525361 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" Sep 08 17:12:54.527764 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.527737 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f0d25d-serving-cert\"" Sep 08 17:12:54.532152 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.532123 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp"] Sep 08 17:12:54.550321 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.550294 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:54.560171 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.560142 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b714ff-1091-41d9-a964-f9aba7af99a0-openshift-service-ca\") pod \"maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp\" (UID: \"02b714ff-1091-41d9-a964-f9aba7af99a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" Sep 08 17:12:54.560283 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.560195 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02b714ff-1091-41d9-a964-f9aba7af99a0-metrics-certs\") pod \"maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp\" (UID: \"02b714ff-1091-41d9-a964-f9aba7af99a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" Sep 08 17:12:54.560326 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.560277 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02b714ff-1091-41d9-a964-f9aba7af99a0-maas-api-tls\") pod \"maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp\" (UID: \"02b714ff-1091-41d9-a964-f9aba7af99a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" Sep 08 17:12:54.560326 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.560319 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csjz9\" (UniqueName: \"kubernetes.io/projected/02b714ff-1091-41d9-a964-f9aba7af99a0-kube-api-access-csjz9\") pod \"maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp\" (UID: \"02b714ff-1091-41d9-a964-f9aba7af99a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" Sep 08 17:12:54.615194 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.615166 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:54.661277 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.661247 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b714ff-1091-41d9-a964-f9aba7af99a0-openshift-service-ca\") pod \"maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp\" (UID: \"02b714ff-1091-41d9-a964-f9aba7af99a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" Sep 08 17:12:54.661469 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.661290 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02b714ff-1091-41d9-a964-f9aba7af99a0-metrics-certs\") pod \"maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp\" (UID: \"02b714ff-1091-41d9-a964-f9aba7af99a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" Sep 08 17:12:54.661469 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.661341 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02b714ff-1091-41d9-a964-f9aba7af99a0-maas-api-tls\") pod \"maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp\" (UID: \"02b714ff-1091-41d9-a964-f9aba7af99a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" Sep 08 17:12:54.661469 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.661368 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csjz9\" (UniqueName: \"kubernetes.io/projected/02b714ff-1091-41d9-a964-f9aba7af99a0-kube-api-access-csjz9\") pod \"maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp\" (UID: \"02b714ff-1091-41d9-a964-f9aba7af99a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" Sep 08 17:12:54.661973 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.661950 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b714ff-1091-41d9-a964-f9aba7af99a0-openshift-service-ca\") pod \"maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp\" (UID: \"02b714ff-1091-41d9-a964-f9aba7af99a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" Sep 08 17:12:54.663778 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.663755 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02b714ff-1091-41d9-a964-f9aba7af99a0-maas-api-tls\") pod \"maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp\" (UID: \"02b714ff-1091-41d9-a964-f9aba7af99a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" Sep 08 17:12:54.663907 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.663888 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02b714ff-1091-41d9-a964-f9aba7af99a0-metrics-certs\") pod \"maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp\" (UID: \"02b714ff-1091-41d9-a964-f9aba7af99a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" Sep 08 17:12:54.675057 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.675032 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csjz9\" (UniqueName: \"kubernetes.io/projected/02b714ff-1091-41d9-a964-f9aba7af99a0-kube-api-access-csjz9\") pod \"maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp\" (UID: \"02b714ff-1091-41d9-a964-f9aba7af99a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" Sep 08 17:12:54.717942 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.717915 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:54.839678 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:54.839598 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" Sep 08 17:12:55.063769 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:55.063734 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:55.130466 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:55.130433 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp"] Sep 08 17:12:55.133109 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:12:55.133074 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b714ff_1091_41d9_a964_f9aba7af99a0.slice/crio-e7d316193ad0ebd574bf9867f3f73774ada9c4ecbbd2b21ea9548bf836ef4031 WatchSource:0}: Error finding container e7d316193ad0ebd574bf9867f3f73774ada9c4ecbbd2b21ea9548bf836ef4031: Status 404 returned error can't find the container with id e7d316193ad0ebd574bf9867f3f73774ada9c4ecbbd2b21ea9548bf836ef4031 Sep 08 17:12:55.134638 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:55.134618 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 17:12:55.161048 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:55.161014 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:12:55.414118 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:55.414031 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" event={"ID":"02b714ff-1091-41d9-a964-f9aba7af99a0","Type":"ContainerStarted","Data":"e7d316193ad0ebd574bf9867f3f73774ada9c4ecbbd2b21ea9548bf836ef4031"} Sep 08 17:12:56.422771 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:56.422734 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" event={"ID":"02b714ff-1091-41d9-a964-f9aba7af99a0","Type":"ContainerStarted","Data":"21f105ce2bb70ca7eb1efb01b8f0357384f3746ba91e3efb2ab56154338b1f63"} Sep 08 17:12:56.423174 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:56.422800 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" Sep 08 17:12:56.442941 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:56.442894 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" podStartSLOduration=1.879980904 podStartE2EDuration="2.442879521s" totalImagesPullingTime="562.898617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:12:56.439465313 +0000 UTC m=+1339.176296428" watchObservedRunningTime="2026-09-08 17:12:56.442879521 +0000 UTC m=+1339.179710672" Sep 08 17:12:59.293961 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:12:59.293925 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:13:02.434928 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:02.434848 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" Sep 08 17:13:18.205892 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:18.205853 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:13:18.669813 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:18.669773 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:13:18.999438 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:18.999320 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:13:19.418573 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:19.418538 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:13:22.498585 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:22.498539 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:13:22.596896 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:22.596837 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:13:23.616748 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:23.616716 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk"] Sep 08 17:13:23.620748 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:23.620718 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" Sep 08 17:13:23.623132 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:23.623108 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b9f2b8-serving-cert\"" Sep 08 17:13:23.632629 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:23.632603 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk"] Sep 08 17:13:23.653329 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:23.653298 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:13:23.729492 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:23.729455 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d921345b-61ec-4a47-8305-049f2e94e464-openshift-service-ca\") pod \"maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk\" (UID: \"d921345b-61ec-4a47-8305-049f2e94e464\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" Sep 08 17:13:23.729653 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:23.729510 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrfc9\" (UniqueName: \"kubernetes.io/projected/d921345b-61ec-4a47-8305-049f2e94e464-kube-api-access-rrfc9\") pod \"maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk\" (UID: \"d921345b-61ec-4a47-8305-049f2e94e464\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" Sep 08 17:13:23.729653 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:23.729545 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d921345b-61ec-4a47-8305-049f2e94e464-maas-api-tls\") pod \"maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk\" (UID: \"d921345b-61ec-4a47-8305-049f2e94e464\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" Sep 08 17:13:23.729653 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:23.729559 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d921345b-61ec-4a47-8305-049f2e94e464-metrics-certs\") pod \"maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk\" (UID: \"d921345b-61ec-4a47-8305-049f2e94e464\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" Sep 08 17:13:23.754689 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:23.754661 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:13:23.830086 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:23.830054 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrfc9\" (UniqueName: \"kubernetes.io/projected/d921345b-61ec-4a47-8305-049f2e94e464-kube-api-access-rrfc9\") pod \"maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk\" (UID: \"d921345b-61ec-4a47-8305-049f2e94e464\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" Sep 08 17:13:23.830229 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:23.830100 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d921345b-61ec-4a47-8305-049f2e94e464-maas-api-tls\") pod \"maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk\" (UID: \"d921345b-61ec-4a47-8305-049f2e94e464\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" Sep 08 17:13:23.830229 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:23.830128 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d921345b-61ec-4a47-8305-049f2e94e464-metrics-certs\") pod \"maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk\" (UID: \"d921345b-61ec-4a47-8305-049f2e94e464\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" Sep 08 17:13:23.830330 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:13:23.830246 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-serving-cert: secret "maas-api-e2e-api-b-b9f2b8-serving-cert" not found Sep 08 17:13:23.830330 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:23.830274 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d921345b-61ec-4a47-8305-049f2e94e464-openshift-service-ca\") pod \"maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk\" (UID: \"d921345b-61ec-4a47-8305-049f2e94e464\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" Sep 08 17:13:23.830330 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:13:23.830321 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d921345b-61ec-4a47-8305-049f2e94e464-maas-api-tls podName:d921345b-61ec-4a47-8305-049f2e94e464 nodeName:}" failed. No retries permitted until 2026-09-08 17:13:24.330302113 +0000 UTC m=+1367.067133222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d921345b-61ec-4a47-8305-049f2e94e464-maas-api-tls") pod "maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" (UID: "d921345b-61ec-4a47-8305-049f2e94e464") : secret "maas-api-e2e-api-b-b9f2b8-serving-cert" not found Sep 08 17:13:23.830943 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:23.830923 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d921345b-61ec-4a47-8305-049f2e94e464-openshift-service-ca\") pod \"maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk\" (UID: \"d921345b-61ec-4a47-8305-049f2e94e464\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" Sep 08 17:13:23.832797 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:23.832775 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d921345b-61ec-4a47-8305-049f2e94e464-metrics-certs\") pod \"maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk\" (UID: \"d921345b-61ec-4a47-8305-049f2e94e464\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" Sep 08 17:13:23.838695 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:23.838673 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrfc9\" (UniqueName: \"kubernetes.io/projected/d921345b-61ec-4a47-8305-049f2e94e464-kube-api-access-rrfc9\") pod \"maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk\" (UID: \"d921345b-61ec-4a47-8305-049f2e94e464\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" Sep 08 17:13:24.024080 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:24.024049 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:13:24.255351 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:24.255315 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:13:24.335323 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:24.335222 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d921345b-61ec-4a47-8305-049f2e94e464-maas-api-tls\") pod \"maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk\" (UID: \"d921345b-61ec-4a47-8305-049f2e94e464\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" Sep 08 17:13:24.337677 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:24.337643 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d921345b-61ec-4a47-8305-049f2e94e464-maas-api-tls\") pod \"maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk\" (UID: \"d921345b-61ec-4a47-8305-049f2e94e464\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" Sep 08 17:13:24.356573 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:24.356538 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:13:24.533635 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:24.533577 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" Sep 08 17:13:24.814357 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:24.814328 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk"] Sep 08 17:13:24.815490 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:13:24.815462 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd921345b_61ec_4a47_8305_049f2e94e464.slice/crio-4de1da882e45ea9a4deb554b78407915e1453b13bf175c4a8bf16e5dc3f37140 WatchSource:0}: Error finding container 4de1da882e45ea9a4deb554b78407915e1453b13bf175c4a8bf16e5dc3f37140: Status 404 returned error can't find the container with id 4de1da882e45ea9a4deb554b78407915e1453b13bf175c4a8bf16e5dc3f37140 Sep 08 17:13:25.604912 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:25.604829 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" event={"ID":"d921345b-61ec-4a47-8305-049f2e94e464","Type":"ContainerStarted","Data":"1397eec58d65baeb936f63314307ae852509306cf4db2e1ca33e30c36992086f"} Sep 08 17:13:25.604912 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:25.604886 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" event={"ID":"d921345b-61ec-4a47-8305-049f2e94e464","Type":"ContainerStarted","Data":"4de1da882e45ea9a4deb554b78407915e1453b13bf175c4a8bf16e5dc3f37140"} Sep 08 17:13:25.605096 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:25.605040 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" Sep 08 17:13:25.620944 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:25.620887 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" podStartSLOduration=2.160438239 podStartE2EDuration="2.620868436s" totalImagesPullingTime="460.430197ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:13:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:13:25.620040522 +0000 UTC m=+1368.356871637" watchObservedRunningTime="2026-09-08 17:13:25.620868436 +0000 UTC m=+1368.357699556" Sep 08 17:13:29.439710 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:29.439669 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:13:31.617500 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:31.617470 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" Sep 08 17:13:47.356052 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:47.356014 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:13:50.540967 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:50.540919 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:13:51.285025 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:51.284976 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:13:52.030384 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:52.030349 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:13:55.733048 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:13:55.733012 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:00.511532 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:00.511486 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:03.397194 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:03.397149 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:06.171524 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.171486 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp"] Sep 08 17:14:06.171991 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.171748 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" podUID="02b714ff-1091-41d9-a964-f9aba7af99a0" containerName="maas-api" containerID="cri-o://21f105ce2bb70ca7eb1efb01b8f0357384f3746ba91e3efb2ab56154338b1f63" gracePeriod=30 Sep 08 17:14:06.208154 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.208126 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:06.274886 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.274838 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:06.337999 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.337957 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:06.413092 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.413061 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:06.432637 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.432583 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" Sep 08 17:14:06.515081 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.515046 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02b714ff-1091-41d9-a964-f9aba7af99a0-metrics-certs\") pod \"02b714ff-1091-41d9-a964-f9aba7af99a0\" (UID: \"02b714ff-1091-41d9-a964-f9aba7af99a0\") " Sep 08 17:14:06.515257 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.515238 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csjz9\" (UniqueName: \"kubernetes.io/projected/02b714ff-1091-41d9-a964-f9aba7af99a0-kube-api-access-csjz9\") pod \"02b714ff-1091-41d9-a964-f9aba7af99a0\" (UID: \"02b714ff-1091-41d9-a964-f9aba7af99a0\") " Sep 08 17:14:06.515361 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.515346 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b714ff-1091-41d9-a964-f9aba7af99a0-openshift-service-ca\") pod \"02b714ff-1091-41d9-a964-f9aba7af99a0\" (UID: \"02b714ff-1091-41d9-a964-f9aba7af99a0\") " Sep 08 17:14:06.515433 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.515375 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02b714ff-1091-41d9-a964-f9aba7af99a0-maas-api-tls\") pod \"02b714ff-1091-41d9-a964-f9aba7af99a0\" (UID: \"02b714ff-1091-41d9-a964-f9aba7af99a0\") " Sep 08 17:14:06.515866 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.515833 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02b714ff-1091-41d9-a964-f9aba7af99a0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "02b714ff-1091-41d9-a964-f9aba7af99a0" (UID: "02b714ff-1091-41d9-a964-f9aba7af99a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:14:06.517462 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.517436 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b714ff-1091-41d9-a964-f9aba7af99a0-kube-api-access-csjz9" (OuterVolumeSpecName: "kube-api-access-csjz9") pod "02b714ff-1091-41d9-a964-f9aba7af99a0" (UID: "02b714ff-1091-41d9-a964-f9aba7af99a0"). InnerVolumeSpecName "kube-api-access-csjz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:14:06.517745 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.517728 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02b714ff-1091-41d9-a964-f9aba7af99a0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "02b714ff-1091-41d9-a964-f9aba7af99a0" (UID: "02b714ff-1091-41d9-a964-f9aba7af99a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:14:06.517820 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.517753 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02b714ff-1091-41d9-a964-f9aba7af99a0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "02b714ff-1091-41d9-a964-f9aba7af99a0" (UID: "02b714ff-1091-41d9-a964-f9aba7af99a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:14:06.616188 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.616151 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b714ff-1091-41d9-a964-f9aba7af99a0-openshift-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:14:06.616188 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.616182 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02b714ff-1091-41d9-a964-f9aba7af99a0-maas-api-tls\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:14:06.616188 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.616192 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02b714ff-1091-41d9-a964-f9aba7af99a0-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:14:06.616449 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.616201 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csjz9\" (UniqueName: \"kubernetes.io/projected/02b714ff-1091-41d9-a964-f9aba7af99a0-kube-api-access-csjz9\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:14:06.750138 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.749964 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:06.858477 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.858438 2565 generic.go:356] "Generic (PLEG): container finished" podID="02b714ff-1091-41d9-a964-f9aba7af99a0" containerID="21f105ce2bb70ca7eb1efb01b8f0357384f3746ba91e3efb2ab56154338b1f63" exitCode=0 Sep 08 17:14:06.858671 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.858486 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" event={"ID":"02b714ff-1091-41d9-a964-f9aba7af99a0","Type":"ContainerDied","Data":"21f105ce2bb70ca7eb1efb01b8f0357384f3746ba91e3efb2ab56154338b1f63"} Sep 08 17:14:06.858671 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.858563 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" event={"ID":"02b714ff-1091-41d9-a964-f9aba7af99a0","Type":"ContainerDied","Data":"e7d316193ad0ebd574bf9867f3f73774ada9c4ecbbd2b21ea9548bf836ef4031"} Sep 08 17:14:06.858671 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.858498 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp" Sep 08 17:14:06.858671 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.858595 2565 scope.go:117] "RemoveContainer" containerID="21f105ce2bb70ca7eb1efb01b8f0357384f3746ba91e3efb2ab56154338b1f63" Sep 08 17:14:06.872348 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.872327 2565 scope.go:117] "RemoveContainer" containerID="21f105ce2bb70ca7eb1efb01b8f0357384f3746ba91e3efb2ab56154338b1f63" Sep 08 17:14:06.872661 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:14:06.872639 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21f105ce2bb70ca7eb1efb01b8f0357384f3746ba91e3efb2ab56154338b1f63\": container with ID starting with 21f105ce2bb70ca7eb1efb01b8f0357384f3746ba91e3efb2ab56154338b1f63 not found: ID does not exist" containerID="21f105ce2bb70ca7eb1efb01b8f0357384f3746ba91e3efb2ab56154338b1f63" Sep 08 17:14:06.872720 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.872670 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21f105ce2bb70ca7eb1efb01b8f0357384f3746ba91e3efb2ab56154338b1f63"} err="failed to get container status \"21f105ce2bb70ca7eb1efb01b8f0357384f3746ba91e3efb2ab56154338b1f63\": rpc error: code = NotFound desc = could not find container \"21f105ce2bb70ca7eb1efb01b8f0357384f3746ba91e3efb2ab56154338b1f63\": container with ID starting with 21f105ce2bb70ca7eb1efb01b8f0357384f3746ba91e3efb2ab56154338b1f63 not found: ID does not exist" Sep 08 17:14:06.881834 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.881812 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp"] Sep 08 17:14:06.887162 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:06.887141 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f0d25d-74d9c97697-fkqfp"] Sep 08 17:14:07.762448 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:07.762392 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02b714ff-1091-41d9-a964-f9aba7af99a0" path="/var/lib/kubelet/pods/02b714ff-1091-41d9-a964-f9aba7af99a0/volumes" Sep 08 17:14:12.732507 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:12.732458 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:12.832652 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:12.832596 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:13.001110 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:13.001018 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:13.541922 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:13.541884 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:14.849800 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:14.849764 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7832-c7e353"] Sep 08 17:14:14.850196 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:14.850183 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02b714ff-1091-41d9-a964-f9aba7af99a0" containerName="maas-api" Sep 08 17:14:14.850246 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:14.850198 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="02b714ff-1091-41d9-a964-f9aba7af99a0" containerName="maas-api" Sep 08 17:14:14.850283 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:14.850278 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="02b714ff-1091-41d9-a964-f9aba7af99a0" containerName="maas-api" Sep 08 17:14:14.853658 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:14.853642 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-c7e353" Sep 08 17:14:14.856378 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:14.856352 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v8zrb\"" Sep 08 17:14:14.859200 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:14.859169 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7832-c7e353"] Sep 08 17:14:14.993079 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:14.993044 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fww5t\" (UniqueName: \"kubernetes.io/projected/4653cc73-7376-4db8-b831-120276df043e-kube-api-access-fww5t\") pod \"test-curl-7832-c7e353\" (UID: \"4653cc73-7376-4db8-b831-120276df043e\") " pod="opendatahub/test-curl-7832-c7e353" Sep 08 17:14:15.094540 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:15.094490 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fww5t\" (UniqueName: \"kubernetes.io/projected/4653cc73-7376-4db8-b831-120276df043e-kube-api-access-fww5t\") pod \"test-curl-7832-c7e353\" (UID: \"4653cc73-7376-4db8-b831-120276df043e\") " pod="opendatahub/test-curl-7832-c7e353" Sep 08 17:14:15.103008 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:15.102934 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fww5t\" (UniqueName: \"kubernetes.io/projected/4653cc73-7376-4db8-b831-120276df043e-kube-api-access-fww5t\") pod \"test-curl-7832-c7e353\" (UID: \"4653cc73-7376-4db8-b831-120276df043e\") " pod="opendatahub/test-curl-7832-c7e353" Sep 08 17:14:15.165391 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:15.165356 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-c7e353" Sep 08 17:14:15.436525 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:15.436499 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7832-c7e353"] Sep 08 17:14:15.438675 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:14:15.438645 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4653cc73_7376_4db8_b831_120276df043e.slice/crio-d4768930958d433e83bc3e2be72ea1f5fadc16a38000535848854b2b13d73bb8 WatchSource:0}: Error finding container d4768930958d433e83bc3e2be72ea1f5fadc16a38000535848854b2b13d73bb8: Status 404 returned error can't find the container with id d4768930958d433e83bc3e2be72ea1f5fadc16a38000535848854b2b13d73bb8 Sep 08 17:14:15.913705 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:15.913666 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7832-c7e353" event={"ID":"4653cc73-7376-4db8-b831-120276df043e","Type":"ContainerStarted","Data":"19ab575ccc25d724cb00b4138d3df77e517978ddfdd09d6007fe5bb5419d3635"} Sep 08 17:14:15.914063 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:15.913715 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7832-c7e353" event={"ID":"4653cc73-7376-4db8-b831-120276df043e","Type":"ContainerStarted","Data":"d4768930958d433e83bc3e2be72ea1f5fadc16a38000535848854b2b13d73bb8"} Sep 08 17:14:15.927519 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:15.927470 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7832-c7e353" podStartSLOduration=1.927452822 podStartE2EDuration="1.927452822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:14:15.92605713 +0000 UTC m=+1418.662888244" watchObservedRunningTime="2026-09-08 17:14:15.927452822 +0000 UTC m=+1418.664283934" Sep 08 17:14:16.490401 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:16.490366 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7832-c7e353"] Sep 08 17:14:16.494710 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:16.494679 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7832-c7e353"] Sep 08 17:14:16.607774 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:16.607741 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7832-2a8f41"] Sep 08 17:14:16.608177 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:16.608164 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4653cc73-7376-4db8-b831-120276df043e" containerName="test-curl-7832-c7e353" Sep 08 17:14:16.608235 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:16.608178 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4653cc73-7376-4db8-b831-120276df043e" containerName="test-curl-7832-c7e353" Sep 08 17:14:16.608272 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:16.608262 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4653cc73-7376-4db8-b831-120276df043e" containerName="test-curl-7832-c7e353" Sep 08 17:14:16.611497 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:16.611478 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-2a8f41" Sep 08 17:14:16.618707 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:16.618681 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7832-2a8f41"] Sep 08 17:14:16.710462 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:16.710406 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9ldj\" (UniqueName: \"kubernetes.io/projected/23de532a-437e-40e7-98f6-e232acd1957c-kube-api-access-q9ldj\") pod \"test-curl-7832-2a8f41\" (UID: \"23de532a-437e-40e7-98f6-e232acd1957c\") " pod="opendatahub/test-curl-7832-2a8f41" Sep 08 17:14:16.811869 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:16.811783 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9ldj\" (UniqueName: \"kubernetes.io/projected/23de532a-437e-40e7-98f6-e232acd1957c-kube-api-access-q9ldj\") pod \"test-curl-7832-2a8f41\" (UID: \"23de532a-437e-40e7-98f6-e232acd1957c\") " pod="opendatahub/test-curl-7832-2a8f41" Sep 08 17:14:16.819826 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:16.819801 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9ldj\" (UniqueName: \"kubernetes.io/projected/23de532a-437e-40e7-98f6-e232acd1957c-kube-api-access-q9ldj\") pod \"test-curl-7832-2a8f41\" (UID: \"23de532a-437e-40e7-98f6-e232acd1957c\") " pod="opendatahub/test-curl-7832-2a8f41" Sep 08 17:14:16.919917 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:14:16.919889 2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7832-c7e353" container="test-curl-7832-c7e353" Sep 08 17:14:16.923766 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:16.923751 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-2a8f41" Sep 08 17:14:17.196445 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:17.196398 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7832-2a8f41"] Sep 08 17:14:17.197492 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:14:17.197459 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23de532a_437e_40e7_98f6_e232acd1957c.slice/crio-7ea89869ddd314797ef410b7db214a70bc4171d12190a58eb7ba33f3924c7a2a WatchSource:0}: Error finding container 7ea89869ddd314797ef410b7db214a70bc4171d12190a58eb7ba33f3924c7a2a: Status 404 returned error can't find the container with id 7ea89869ddd314797ef410b7db214a70bc4171d12190a58eb7ba33f3924c7a2a Sep 08 17:14:17.927726 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:17.927691 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7832-2a8f41" event={"ID":"23de532a-437e-40e7-98f6-e232acd1957c","Type":"ContainerStarted","Data":"6967275b7e29ee8491dcb3cd4562dd7dddf2fe537a4b363e463537b18a9d218b"} Sep 08 17:14:17.927726 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:17.927732 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7832-2a8f41" event={"ID":"23de532a-437e-40e7-98f6-e232acd1957c","Type":"ContainerStarted","Data":"7ea89869ddd314797ef410b7db214a70bc4171d12190a58eb7ba33f3924c7a2a"} Sep 08 17:14:17.928210 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:17.927851 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7832-c7e353" podUID="4653cc73-7376-4db8-b831-120276df043e" containerName="test-curl-7832-c7e353" containerID="cri-o://19ab575ccc25d724cb00b4138d3df77e517978ddfdd09d6007fe5bb5419d3635" gracePeriod=2 Sep 08 17:14:17.942725 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:17.942673 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7832-2a8f41" podStartSLOduration=1.942658914 podStartE2EDuration="1.942658914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:14:17.942087229 +0000 UTC m=+1420.678918344" watchObservedRunningTime="2026-09-08 17:14:17.942658914 +0000 UTC m=+1420.679490029" Sep 08 17:14:18.516300 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:18.516264 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7832-2a8f41"] Sep 08 17:14:18.520209 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:18.520175 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7832-2a8f41"] Sep 08 17:14:18.933595 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:14:18.933567 2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7832-2a8f41" container="test-curl-7832-2a8f41" Sep 08 17:14:19.930358 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:19.930320 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:19.941669 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:19.941637 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7832-2a8f41" podUID="23de532a-437e-40e7-98f6-e232acd1957c" containerName="test-curl-7832-2a8f41" containerID="cri-o://6967275b7e29ee8491dcb3cd4562dd7dddf2fe537a4b363e463537b18a9d218b" gracePeriod=2 Sep 08 17:14:20.077385 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:20.077356 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-c7e353" Sep 08 17:14:20.243731 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:20.243652 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fww5t\" (UniqueName: \"kubernetes.io/projected/4653cc73-7376-4db8-b831-120276df043e-kube-api-access-fww5t\") pod \"4653cc73-7376-4db8-b831-120276df043e\" (UID: \"4653cc73-7376-4db8-b831-120276df043e\") " Sep 08 17:14:20.245654 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:20.245622 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4653cc73-7376-4db8-b831-120276df043e-kube-api-access-fww5t" (OuterVolumeSpecName: "kube-api-access-fww5t") pod "4653cc73-7376-4db8-b831-120276df043e" (UID: "4653cc73-7376-4db8-b831-120276df043e"). InnerVolumeSpecName "kube-api-access-fww5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:14:20.344870 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:20.344835 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fww5t\" (UniqueName: \"kubernetes.io/projected/4653cc73-7376-4db8-b831-120276df043e-kube-api-access-fww5t\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:14:20.517820 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:20.517730 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:20.948694 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:20.948657 2565 generic.go:356] "Generic (PLEG): container finished" podID="4653cc73-7376-4db8-b831-120276df043e" containerID="19ab575ccc25d724cb00b4138d3df77e517978ddfdd09d6007fe5bb5419d3635" exitCode=137 Sep 08 17:14:20.949172 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:20.948699 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-c7e353" Sep 08 17:14:20.949172 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:20.948778 2565 scope.go:117] "RemoveContainer" containerID="19ab575ccc25d724cb00b4138d3df77e517978ddfdd09d6007fe5bb5419d3635" Sep 08 17:14:20.959279 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:20.959260 2565 scope.go:117] "RemoveContainer" containerID="19ab575ccc25d724cb00b4138d3df77e517978ddfdd09d6007fe5bb5419d3635" Sep 08 17:14:20.959484 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:14:20.959461 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7832-c7e353\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="4653cc73-7376-4db8-b831-120276df043e" pod="opendatahub/test-curl-7832-c7e353" Sep 08 17:14:20.959568 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:14:20.959552 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ab575ccc25d724cb00b4138d3df77e517978ddfdd09d6007fe5bb5419d3635\": container with ID starting with 19ab575ccc25d724cb00b4138d3df77e517978ddfdd09d6007fe5bb5419d3635 not found: ID does not exist" containerID="19ab575ccc25d724cb00b4138d3df77e517978ddfdd09d6007fe5bb5419d3635" Sep 08 17:14:20.959611 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:20.959577 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ab575ccc25d724cb00b4138d3df77e517978ddfdd09d6007fe5bb5419d3635"} err="failed to get container status \"19ab575ccc25d724cb00b4138d3df77e517978ddfdd09d6007fe5bb5419d3635\": rpc error: code = NotFound desc = could not find container \"19ab575ccc25d724cb00b4138d3df77e517978ddfdd09d6007fe5bb5419d3635\": container with ID starting with 19ab575ccc25d724cb00b4138d3df77e517978ddfdd09d6007fe5bb5419d3635 not found: ID does not exist" Sep 08 17:14:21.607616 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:21.607580 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:21.761128 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:21.761098 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4653cc73-7376-4db8-b831-120276df043e" path="/var/lib/kubelet/pods/4653cc73-7376-4db8-b831-120276df043e/volumes" Sep 08 17:14:22.098115 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:22.098085 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-2a8f41" Sep 08 17:14:22.260416 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:22.260336 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9ldj\" (UniqueName: \"kubernetes.io/projected/23de532a-437e-40e7-98f6-e232acd1957c-kube-api-access-q9ldj\") pod \"23de532a-437e-40e7-98f6-e232acd1957c\" (UID: \"23de532a-437e-40e7-98f6-e232acd1957c\") " Sep 08 17:14:22.262355 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:22.262328 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23de532a-437e-40e7-98f6-e232acd1957c-kube-api-access-q9ldj" (OuterVolumeSpecName: "kube-api-access-q9ldj") pod "23de532a-437e-40e7-98f6-e232acd1957c" (UID: "23de532a-437e-40e7-98f6-e232acd1957c"). InnerVolumeSpecName "kube-api-access-q9ldj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:14:22.361312 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:22.361281 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9ldj\" (UniqueName: \"kubernetes.io/projected/23de532a-437e-40e7-98f6-e232acd1957c-kube-api-access-q9ldj\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:14:22.966018 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:22.965982 2565 generic.go:356] "Generic (PLEG): container finished" podID="23de532a-437e-40e7-98f6-e232acd1957c" containerID="6967275b7e29ee8491dcb3cd4562dd7dddf2fe537a4b363e463537b18a9d218b" exitCode=137 Sep 08 17:14:22.966018 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:22.966019 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-2a8f41" Sep 08 17:14:22.966274 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:22.966079 2565 scope.go:117] "RemoveContainer" containerID="6967275b7e29ee8491dcb3cd4562dd7dddf2fe537a4b363e463537b18a9d218b" Sep 08 17:14:22.975938 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:22.975919 2565 scope.go:117] "RemoveContainer" containerID="6967275b7e29ee8491dcb3cd4562dd7dddf2fe537a4b363e463537b18a9d218b" Sep 08 17:14:22.976156 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:14:22.976130 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7832-2a8f41\" is forbidden: User \"system:node:ip-10-0-137-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-108.ec2.internal' and this object" podUID="23de532a-437e-40e7-98f6-e232acd1957c" pod="opendatahub/test-curl-7832-2a8f41" Sep 08 17:14:22.976213 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:14:22.976167 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6967275b7e29ee8491dcb3cd4562dd7dddf2fe537a4b363e463537b18a9d218b\": container with ID starting with 6967275b7e29ee8491dcb3cd4562dd7dddf2fe537a4b363e463537b18a9d218b not found: ID does not exist" containerID="6967275b7e29ee8491dcb3cd4562dd7dddf2fe537a4b363e463537b18a9d218b" Sep 08 17:14:22.976213 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:22.976190 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6967275b7e29ee8491dcb3cd4562dd7dddf2fe537a4b363e463537b18a9d218b"} err="failed to get container status \"6967275b7e29ee8491dcb3cd4562dd7dddf2fe537a4b363e463537b18a9d218b\": rpc error: code = NotFound desc = could not find container \"6967275b7e29ee8491dcb3cd4562dd7dddf2fe537a4b363e463537b18a9d218b\": container with ID starting with 6967275b7e29ee8491dcb3cd4562dd7dddf2fe537a4b363e463537b18a9d218b not found: ID does not exist" Sep 08 17:14:23.447789 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:23.447754 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:23.761111 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:23.761032 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23de532a-437e-40e7-98f6-e232acd1957c" path="/var/lib/kubelet/pods/23de532a-437e-40e7-98f6-e232acd1957c/volumes" Sep 08 17:14:25.337965 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:25.337923 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:27.202641 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:27.202604 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:29.101465 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:29.101417 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk"] Sep 08 17:14:29.101836 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:29.101639 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" podUID="d921345b-61ec-4a47-8305-049f2e94e464" containerName="maas-api" containerID="cri-o://1397eec58d65baeb936f63314307ae852509306cf4db2e1ca33e30c36992086f" gracePeriod=30 Sep 08 17:14:29.116302 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:29.116221 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:29.220733 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:29.220702 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:29.317163 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:29.317125 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:29.356818 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:29.356766 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" Sep 08 17:14:29.419315 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:29.419286 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:29.420227 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:29.420208 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d921345b-61ec-4a47-8305-049f2e94e464-openshift-service-ca\") pod \"d921345b-61ec-4a47-8305-049f2e94e464\" (UID: \"d921345b-61ec-4a47-8305-049f2e94e464\") " Sep 08 17:14:29.420323 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:29.420238 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrfc9\" (UniqueName: \"kubernetes.io/projected/d921345b-61ec-4a47-8305-049f2e94e464-kube-api-access-rrfc9\") pod \"d921345b-61ec-4a47-8305-049f2e94e464\" (UID: \"d921345b-61ec-4a47-8305-049f2e94e464\") " Sep 08 17:14:29.420323 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:29.420268 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d921345b-61ec-4a47-8305-049f2e94e464-maas-api-tls\") pod \"d921345b-61ec-4a47-8305-049f2e94e464\" (UID: \"d921345b-61ec-4a47-8305-049f2e94e464\") " Sep 08 17:14:29.420456 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:29.420362 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d921345b-61ec-4a47-8305-049f2e94e464-metrics-certs\") pod \"d921345b-61ec-4a47-8305-049f2e94e464\" (UID: \"d921345b-61ec-4a47-8305-049f2e94e464\") " Sep 08 17:14:29.420639 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:29.420615 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d921345b-61ec-4a47-8305-049f2e94e464-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d921345b-61ec-4a47-8305-049f2e94e464" (UID: "d921345b-61ec-4a47-8305-049f2e94e464"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:14:29.422402 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:29.422382 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d921345b-61ec-4a47-8305-049f2e94e464-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d921345b-61ec-4a47-8305-049f2e94e464" (UID: "d921345b-61ec-4a47-8305-049f2e94e464"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:14:29.422505 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:29.422409 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d921345b-61ec-4a47-8305-049f2e94e464-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d921345b-61ec-4a47-8305-049f2e94e464" (UID: "d921345b-61ec-4a47-8305-049f2e94e464"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:14:29.422562 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:29.422529 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d921345b-61ec-4a47-8305-049f2e94e464-kube-api-access-rrfc9" (OuterVolumeSpecName: "kube-api-access-rrfc9") pod "d921345b-61ec-4a47-8305-049f2e94e464" (UID: "d921345b-61ec-4a47-8305-049f2e94e464"). InnerVolumeSpecName "kube-api-access-rrfc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:14:29.521261 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:29.521228 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d921345b-61ec-4a47-8305-049f2e94e464-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:14:29.521261 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:29.521259 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d921345b-61ec-4a47-8305-049f2e94e464-openshift-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:14:29.521463 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:29.521269 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrfc9\" (UniqueName: \"kubernetes.io/projected/d921345b-61ec-4a47-8305-049f2e94e464-kube-api-access-rrfc9\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:14:29.521463 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:29.521279 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d921345b-61ec-4a47-8305-049f2e94e464-maas-api-tls\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:14:29.709449 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:29.709394 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:30.010504 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:30.010384 2565 generic.go:356] "Generic (PLEG): container finished" podID="d921345b-61ec-4a47-8305-049f2e94e464" containerID="1397eec58d65baeb936f63314307ae852509306cf4db2e1ca33e30c36992086f" exitCode=0 Sep 08 17:14:30.010504 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:30.010479 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" Sep 08 17:14:30.010681 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:30.010475 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" event={"ID":"d921345b-61ec-4a47-8305-049f2e94e464","Type":"ContainerDied","Data":"1397eec58d65baeb936f63314307ae852509306cf4db2e1ca33e30c36992086f"} Sep 08 17:14:30.010681 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:30.010609 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk" event={"ID":"d921345b-61ec-4a47-8305-049f2e94e464","Type":"ContainerDied","Data":"4de1da882e45ea9a4deb554b78407915e1453b13bf175c4a8bf16e5dc3f37140"} Sep 08 17:14:30.010681 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:30.010633 2565 scope.go:117] "RemoveContainer" containerID="1397eec58d65baeb936f63314307ae852509306cf4db2e1ca33e30c36992086f" Sep 08 17:14:30.019871 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:30.019850 2565 scope.go:117] "RemoveContainer" containerID="1397eec58d65baeb936f63314307ae852509306cf4db2e1ca33e30c36992086f" Sep 08 17:14:30.020138 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:14:30.020117 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1397eec58d65baeb936f63314307ae852509306cf4db2e1ca33e30c36992086f\": container with ID starting with 1397eec58d65baeb936f63314307ae852509306cf4db2e1ca33e30c36992086f not found: ID does not exist" containerID="1397eec58d65baeb936f63314307ae852509306cf4db2e1ca33e30c36992086f" Sep 08 17:14:30.020192 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:30.020146 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1397eec58d65baeb936f63314307ae852509306cf4db2e1ca33e30c36992086f"} err="failed to get container status \"1397eec58d65baeb936f63314307ae852509306cf4db2e1ca33e30c36992086f\": rpc error: code = NotFound desc = could not find container \"1397eec58d65baeb936f63314307ae852509306cf4db2e1ca33e30c36992086f\": container with ID starting with 1397eec58d65baeb936f63314307ae852509306cf4db2e1ca33e30c36992086f not found: ID does not exist" Sep 08 17:14:30.032341 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:30.032318 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk"] Sep 08 17:14:30.035683 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:30.035663 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9f2b8-8bbb99754-rgmhk"] Sep 08 17:14:31.761222 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:31.761177 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d921345b-61ec-4a47-8305-049f2e94e464" path="/var/lib/kubelet/pods/d921345b-61ec-4a47-8305-049f2e94e464/volumes" Sep 08 17:14:35.877480 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:35.877416 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:35.977798 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:35.977758 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:36.149013 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:36.148927 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:36.607456 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:36.607392 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:40.594709 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:40.594603 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:46.934034 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:46.933999 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg"] Sep 08 17:14:46.934494 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:46.934303 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" podUID="57ed1a95-f79c-4242-a5f2-45eafca7b893" containerName="maas-api" containerID="cri-o://4fbbcfc97a37ecdc1f5e683edfb63c2c226c16b6f7364e00be0ab4c4d566fe77" gracePeriod=30 Sep 08 17:14:46.966225 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:46.966189 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:47.068386 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:47.068338 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:47.125011 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:47.124947 2565 generic.go:356] "Generic (PLEG): container finished" podID="57ed1a95-f79c-4242-a5f2-45eafca7b893" containerID="4fbbcfc97a37ecdc1f5e683edfb63c2c226c16b6f7364e00be0ab4c4d566fe77" exitCode=0 Sep 08 17:14:47.125011 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:47.124991 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" event={"ID":"57ed1a95-f79c-4242-a5f2-45eafca7b893","Type":"ContainerDied","Data":"4fbbcfc97a37ecdc1f5e683edfb63c2c226c16b6f7364e00be0ab4c4d566fe77"} Sep 08 17:14:47.169539 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:47.169509 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:47.189232 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:47.189178 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" Sep 08 17:14:47.284385 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:47.284346 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57ed1a95-f79c-4242-a5f2-45eafca7b893-maas-api-tls\") pod \"57ed1a95-f79c-4242-a5f2-45eafca7b893\" (UID: \"57ed1a95-f79c-4242-a5f2-45eafca7b893\") " Sep 08 17:14:47.284592 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:47.284405 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ed1a95-f79c-4242-a5f2-45eafca7b893-metrics-certs\") pod \"57ed1a95-f79c-4242-a5f2-45eafca7b893\" (UID: \"57ed1a95-f79c-4242-a5f2-45eafca7b893\") " Sep 08 17:14:47.284592 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:47.284496 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ed1a95-f79c-4242-a5f2-45eafca7b893-openshift-service-ca\") pod \"57ed1a95-f79c-4242-a5f2-45eafca7b893\" (UID: \"57ed1a95-f79c-4242-a5f2-45eafca7b893\") " Sep 08 17:14:47.284592 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:47.284539 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xts9\" (UniqueName: \"kubernetes.io/projected/57ed1a95-f79c-4242-a5f2-45eafca7b893-kube-api-access-5xts9\") pod \"57ed1a95-f79c-4242-a5f2-45eafca7b893\" (UID: \"57ed1a95-f79c-4242-a5f2-45eafca7b893\") " Sep 08 17:14:47.284838 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:47.284806 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ed1a95-f79c-4242-a5f2-45eafca7b893-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "57ed1a95-f79c-4242-a5f2-45eafca7b893" (UID: "57ed1a95-f79c-4242-a5f2-45eafca7b893"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:14:47.284958 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:47.284903 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ed1a95-f79c-4242-a5f2-45eafca7b893-openshift-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:14:47.286517 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:47.286487 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ed1a95-f79c-4242-a5f2-45eafca7b893-kube-api-access-5xts9" (OuterVolumeSpecName: "kube-api-access-5xts9") pod "57ed1a95-f79c-4242-a5f2-45eafca7b893" (UID: "57ed1a95-f79c-4242-a5f2-45eafca7b893"). InnerVolumeSpecName "kube-api-access-5xts9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:14:47.286517 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:47.286492 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57ed1a95-f79c-4242-a5f2-45eafca7b893-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "57ed1a95-f79c-4242-a5f2-45eafca7b893" (UID: "57ed1a95-f79c-4242-a5f2-45eafca7b893"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:14:47.286641 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:47.286523 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57ed1a95-f79c-4242-a5f2-45eafca7b893-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "57ed1a95-f79c-4242-a5f2-45eafca7b893" (UID: "57ed1a95-f79c-4242-a5f2-45eafca7b893"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:14:47.291338 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:47.291312 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:47.386188 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:47.386160 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57ed1a95-f79c-4242-a5f2-45eafca7b893-maas-api-tls\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:14:47.386188 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:47.386185 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ed1a95-f79c-4242-a5f2-45eafca7b893-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:14:47.386188 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:47.386194 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xts9\" (UniqueName: \"kubernetes.io/projected/57ed1a95-f79c-4242-a5f2-45eafca7b893-kube-api-access-5xts9\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:14:47.560675 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:47.560586 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:48.132141 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:48.132100 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" event={"ID":"57ed1a95-f79c-4242-a5f2-45eafca7b893","Type":"ContainerDied","Data":"6f6698fe35a7d77057267945791e2d524b52696fe47533dd3db5cc2246b3e019"} Sep 08 17:14:48.132141 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:48.132129 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg" Sep 08 17:14:48.132660 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:48.132180 2565 scope.go:117] "RemoveContainer" containerID="4fbbcfc97a37ecdc1f5e683edfb63c2c226c16b6f7364e00be0ab4c4d566fe77" Sep 08 17:14:48.154486 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:48.154440 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg"] Sep 08 17:14:48.160458 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:48.160417 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec96ff-69bdfd58f4-9l7lg"] Sep 08 17:14:49.761323 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:49.761283 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57ed1a95-f79c-4242-a5f2-45eafca7b893" path="/var/lib/kubelet/pods/57ed1a95-f79c-4242-a5f2-45eafca7b893/volumes" Sep 08 17:14:50.496695 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:50.496663 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:53.252412 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:53.252371 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:53.319220 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:53.319188 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:53.537529 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:53.537407 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:53.978116 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:53.978077 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:56.131323 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:56.131293 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:14:59.026201 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:14:59.026162 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:15:04.149344 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:04.148727 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:15:04.793552 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:04.793501 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m"] Sep 08 17:15:04.793911 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:04.793857 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" podUID="bacb3d2c-1b29-4413-a5db-81288da98dc5" containerName="maas-api" containerID="cri-o://e965505b888ed8e752b53fba8992c23d89bf217b8858f197e507e7e6dafd7cb3" gracePeriod=30 Sep 08 17:15:04.823194 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:04.823162 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:15:04.922869 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:04.922830 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:15:05.025217 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.025164 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:15:05.057509 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.057415 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" Sep 08 17:15:05.125199 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.125163 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:15:05.149180 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.149151 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bacb3d2c-1b29-4413-a5db-81288da98dc5-maas-api-tls\") pod \"bacb3d2c-1b29-4413-a5db-81288da98dc5\" (UID: \"bacb3d2c-1b29-4413-a5db-81288da98dc5\") " Sep 08 17:15:05.149350 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.149207 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bacb3d2c-1b29-4413-a5db-81288da98dc5-openshift-service-ca\") pod \"bacb3d2c-1b29-4413-a5db-81288da98dc5\" (UID: \"bacb3d2c-1b29-4413-a5db-81288da98dc5\") " Sep 08 17:15:05.149350 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.149294 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7tpd\" (UniqueName: \"kubernetes.io/projected/bacb3d2c-1b29-4413-a5db-81288da98dc5-kube-api-access-s7tpd\") pod \"bacb3d2c-1b29-4413-a5db-81288da98dc5\" (UID: \"bacb3d2c-1b29-4413-a5db-81288da98dc5\") " Sep 08 17:15:05.149350 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.149318 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bacb3d2c-1b29-4413-a5db-81288da98dc5-metrics-certs\") pod \"bacb3d2c-1b29-4413-a5db-81288da98dc5\" (UID: \"bacb3d2c-1b29-4413-a5db-81288da98dc5\") " Sep 08 17:15:05.149893 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.149852 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bacb3d2c-1b29-4413-a5db-81288da98dc5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bacb3d2c-1b29-4413-a5db-81288da98dc5" (UID: "bacb3d2c-1b29-4413-a5db-81288da98dc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:15:05.150027 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.150008 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bacb3d2c-1b29-4413-a5db-81288da98dc5-openshift-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:15:05.151986 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.151945 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bacb3d2c-1b29-4413-a5db-81288da98dc5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bacb3d2c-1b29-4413-a5db-81288da98dc5" (UID: "bacb3d2c-1b29-4413-a5db-81288da98dc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:15:05.152093 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.152052 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bacb3d2c-1b29-4413-a5db-81288da98dc5-kube-api-access-s7tpd" (OuterVolumeSpecName: "kube-api-access-s7tpd") pod "bacb3d2c-1b29-4413-a5db-81288da98dc5" (UID: "bacb3d2c-1b29-4413-a5db-81288da98dc5"). InnerVolumeSpecName "kube-api-access-s7tpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:15:05.152147 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.152130 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bacb3d2c-1b29-4413-a5db-81288da98dc5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bacb3d2c-1b29-4413-a5db-81288da98dc5" (UID: "bacb3d2c-1b29-4413-a5db-81288da98dc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:15:05.231228 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.231194 2565 generic.go:356] "Generic (PLEG): container finished" podID="bacb3d2c-1b29-4413-a5db-81288da98dc5" containerID="e965505b888ed8e752b53fba8992c23d89bf217b8858f197e507e7e6dafd7cb3" exitCode=0 Sep 08 17:15:05.231411 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.231237 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" Sep 08 17:15:05.231411 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.231238 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" event={"ID":"bacb3d2c-1b29-4413-a5db-81288da98dc5","Type":"ContainerDied","Data":"e965505b888ed8e752b53fba8992c23d89bf217b8858f197e507e7e6dafd7cb3"} Sep 08 17:15:05.231411 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.231382 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m" event={"ID":"bacb3d2c-1b29-4413-a5db-81288da98dc5","Type":"ContainerDied","Data":"3b9ae4f92a3839d54dc43655368eae8de4f14e7e36bc9b9301073e777ac04460"} Sep 08 17:15:05.231606 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.231433 2565 scope.go:117] "RemoveContainer" containerID="e965505b888ed8e752b53fba8992c23d89bf217b8858f197e507e7e6dafd7cb3" Sep 08 17:15:05.240822 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.240804 2565 scope.go:117] "RemoveContainer" containerID="e965505b888ed8e752b53fba8992c23d89bf217b8858f197e507e7e6dafd7cb3" Sep 08 17:15:05.241100 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:15:05.241074 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e965505b888ed8e752b53fba8992c23d89bf217b8858f197e507e7e6dafd7cb3\": container with ID starting with e965505b888ed8e752b53fba8992c23d89bf217b8858f197e507e7e6dafd7cb3 not found: ID does not exist" containerID="e965505b888ed8e752b53fba8992c23d89bf217b8858f197e507e7e6dafd7cb3" Sep 08 17:15:05.241152 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.241115 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e965505b888ed8e752b53fba8992c23d89bf217b8858f197e507e7e6dafd7cb3"} err="failed to get container status \"e965505b888ed8e752b53fba8992c23d89bf217b8858f197e507e7e6dafd7cb3\": rpc error: code = NotFound desc = could not find container \"e965505b888ed8e752b53fba8992c23d89bf217b8858f197e507e7e6dafd7cb3\": container with ID starting with e965505b888ed8e752b53fba8992c23d89bf217b8858f197e507e7e6dafd7cb3 not found: ID does not exist" Sep 08 17:15:05.251180 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.251153 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7tpd\" (UniqueName: \"kubernetes.io/projected/bacb3d2c-1b29-4413-a5db-81288da98dc5-kube-api-access-s7tpd\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:15:05.251180 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.251180 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bacb3d2c-1b29-4413-a5db-81288da98dc5-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:15:05.251367 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.251190 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bacb3d2c-1b29-4413-a5db-81288da98dc5-maas-api-tls\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:15:05.253380 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.253357 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m"] Sep 08 17:15:05.256013 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.255991 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7359b-84bb4cc966-5x27m"] Sep 08 17:15:05.387456 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.387341 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:15:05.761088 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:05.761044 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bacb3d2c-1b29-4413-a5db-81288da98dc5" path="/var/lib/kubelet/pods/bacb3d2c-1b29-4413-a5db-81288da98dc5/volumes" Sep 08 17:15:11.896191 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:11.896151 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:15:11.972004 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:11.971973 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:15:37.863397 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:37.863365 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/2.log" Sep 08 17:15:37.868771 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:37.868746 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/2.log" Sep 08 17:15:37.871061 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:37.871040 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xxnl_bf53ac63-f864-402b-bf12-45b289e7e13d/ovn-acl-logging/0.log" Sep 08 17:15:37.876756 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:37.876736 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xxnl_bf53ac63-f864-402b-bf12-45b289e7e13d/ovn-acl-logging/0.log" Sep 08 17:15:55.356184 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:55.356144 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:15:55.463502 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:55.463467 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:15:58.378527 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:58.378462 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:15:58.486477 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:58.486383 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:15:59.636003 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.635963 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt"] Sep 08 17:15:59.636534 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.636467 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bacb3d2c-1b29-4413-a5db-81288da98dc5" containerName="maas-api" Sep 08 17:15:59.636534 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.636482 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bacb3d2c-1b29-4413-a5db-81288da98dc5" containerName="maas-api" Sep 08 17:15:59.636534 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.636491 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23de532a-437e-40e7-98f6-e232acd1957c" containerName="test-curl-7832-2a8f41" Sep 08 17:15:59.636534 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.636504 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="23de532a-437e-40e7-98f6-e232acd1957c" containerName="test-curl-7832-2a8f41" Sep 08 17:15:59.636534 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.636514 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57ed1a95-f79c-4242-a5f2-45eafca7b893" containerName="maas-api" Sep 08 17:15:59.636534 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.636519 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="57ed1a95-f79c-4242-a5f2-45eafca7b893" containerName="maas-api" Sep 08 17:15:59.636534 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.636531 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d921345b-61ec-4a47-8305-049f2e94e464" containerName="maas-api" Sep 08 17:15:59.636534 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.636536 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d921345b-61ec-4a47-8305-049f2e94e464" containerName="maas-api" Sep 08 17:15:59.636909 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.636617 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bacb3d2c-1b29-4413-a5db-81288da98dc5" containerName="maas-api" Sep 08 17:15:59.636909 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.636625 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d921345b-61ec-4a47-8305-049f2e94e464" containerName="maas-api" Sep 08 17:15:59.636909 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.636634 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="23de532a-437e-40e7-98f6-e232acd1957c" containerName="test-curl-7832-2a8f41" Sep 08 17:15:59.636909 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.636640 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="57ed1a95-f79c-4242-a5f2-45eafca7b893" containerName="maas-api" Sep 08 17:15:59.641409 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.641382 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" Sep 08 17:15:59.644532 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.644506 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-978af9-serving-cert\"" Sep 08 17:15:59.660048 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.659932 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt"] Sep 08 17:15:59.676453 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.676400 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:15:59.734099 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.734060 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1805bd0-8a18-44e6-975f-198b54cb6fb2-metrics-certs\") pod \"maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt\" (UID: \"c1805bd0-8a18-44e6-975f-198b54cb6fb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" Sep 08 17:15:59.734099 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.734099 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwfx8\" (UniqueName: \"kubernetes.io/projected/c1805bd0-8a18-44e6-975f-198b54cb6fb2-kube-api-access-pwfx8\") pod \"maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt\" (UID: \"c1805bd0-8a18-44e6-975f-198b54cb6fb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" Sep 08 17:15:59.734344 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.734127 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1805bd0-8a18-44e6-975f-198b54cb6fb2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt\" (UID: \"c1805bd0-8a18-44e6-975f-198b54cb6fb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" Sep 08 17:15:59.734344 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.734232 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1805bd0-8a18-44e6-975f-198b54cb6fb2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt\" (UID: \"c1805bd0-8a18-44e6-975f-198b54cb6fb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" Sep 08 17:15:59.789761 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.789720 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:15:59.835146 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.835106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1805bd0-8a18-44e6-975f-198b54cb6fb2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt\" (UID: \"c1805bd0-8a18-44e6-975f-198b54cb6fb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" Sep 08 17:15:59.835330 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.835161 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1805bd0-8a18-44e6-975f-198b54cb6fb2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt\" (UID: \"c1805bd0-8a18-44e6-975f-198b54cb6fb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" Sep 08 17:15:59.835387 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:15:59.835345 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-serving-cert: secret "maas-api-e2e-ipp-a-978af9-serving-cert" not found Sep 08 17:15:59.835463 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.835415 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1805bd0-8a18-44e6-975f-198b54cb6fb2-metrics-certs\") pod \"maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt\" (UID: \"c1805bd0-8a18-44e6-975f-198b54cb6fb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" Sep 08 17:15:59.835529 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:15:59.835489 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1805bd0-8a18-44e6-975f-198b54cb6fb2-maas-api-tls podName:c1805bd0-8a18-44e6-975f-198b54cb6fb2 nodeName:}" failed. No retries permitted until 2026-09-08 17:16:00.335465436 +0000 UTC m=+1523.072296535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c1805bd0-8a18-44e6-975f-198b54cb6fb2-maas-api-tls") pod "maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" (UID: "c1805bd0-8a18-44e6-975f-198b54cb6fb2") : secret "maas-api-e2e-ipp-a-978af9-serving-cert" not found Sep 08 17:15:59.835529 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.835516 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwfx8\" (UniqueName: \"kubernetes.io/projected/c1805bd0-8a18-44e6-975f-198b54cb6fb2-kube-api-access-pwfx8\") pod \"maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt\" (UID: \"c1805bd0-8a18-44e6-975f-198b54cb6fb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" Sep 08 17:15:59.835887 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.835862 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1805bd0-8a18-44e6-975f-198b54cb6fb2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt\" (UID: \"c1805bd0-8a18-44e6-975f-198b54cb6fb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" Sep 08 17:15:59.837912 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.837891 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1805bd0-8a18-44e6-975f-198b54cb6fb2-metrics-certs\") pod \"maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt\" (UID: \"c1805bd0-8a18-44e6-975f-198b54cb6fb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" Sep 08 17:15:59.845001 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.844975 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwfx8\" (UniqueName: \"kubernetes.io/projected/c1805bd0-8a18-44e6-975f-198b54cb6fb2-kube-api-access-pwfx8\") pod \"maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt\" (UID: \"c1805bd0-8a18-44e6-975f-198b54cb6fb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" Sep 08 17:15:59.914828 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:15:59.914727 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:16:00.340246 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:00.340208 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1805bd0-8a18-44e6-975f-198b54cb6fb2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt\" (UID: \"c1805bd0-8a18-44e6-975f-198b54cb6fb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" Sep 08 17:16:00.342739 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:00.342711 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1805bd0-8a18-44e6-975f-198b54cb6fb2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt\" (UID: \"c1805bd0-8a18-44e6-975f-198b54cb6fb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" Sep 08 17:16:00.369333 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:00.369297 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:16:00.471479 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:00.471442 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:16:00.564118 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:00.564081 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" Sep 08 17:16:00.845286 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:00.845248 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt"] Sep 08 17:16:00.846821 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:16:00.846789 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1805bd0_8a18_44e6_975f_198b54cb6fb2.slice/crio-c69ba91da5e1564bb185613e879154154a93ed45cffa823aacd764c8755154b5 WatchSource:0}: Error finding container c69ba91da5e1564bb185613e879154154a93ed45cffa823aacd764c8755154b5: Status 404 returned error can't find the container with id c69ba91da5e1564bb185613e879154154a93ed45cffa823aacd764c8755154b5 Sep 08 17:16:01.582829 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:01.582792 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" event={"ID":"c1805bd0-8a18-44e6-975f-198b54cb6fb2","Type":"ContainerStarted","Data":"c69ba91da5e1564bb185613e879154154a93ed45cffa823aacd764c8755154b5"} Sep 08 17:16:04.603790 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:04.603753 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" event={"ID":"c1805bd0-8a18-44e6-975f-198b54cb6fb2","Type":"ContainerStarted","Data":"3a40b91490d5c11d720db491edd40730d9ed68b6c9b0368cdb76ae59bfd6186f"} Sep 08 17:16:04.604191 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:04.603835 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" Sep 08 17:16:04.623017 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:04.622967 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" podStartSLOduration=2.636625325 podStartE2EDuration="5.622952475s" totalImagesPullingTime="2.98632715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:16:04.620995995 +0000 UTC m=+1527.357827111" watchObservedRunningTime="2026-09-08 17:16:04.622952475 +0000 UTC m=+1527.359783635" Sep 08 17:16:10.615154 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:10.615121 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" Sep 08 17:16:23.332767 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:23.332728 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:16:24.023167 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:24.023131 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:16:24.371518 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:24.371408 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:16:24.676290 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:24.676249 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:16:27.543130 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:27.543080 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:16:27.630783 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:27.630744 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:16:28.660558 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:28.660524 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc"] Sep 08 17:16:28.664292 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:28.664271 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" Sep 08 17:16:28.666599 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:28.666573 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a78a1e-serving-cert\"" Sep 08 17:16:28.679262 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:28.679225 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc"] Sep 08 17:16:28.703185 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:28.703145 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:16:28.802257 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:28.802212 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa54e18d-d0ff-4c61-8377-4891f35aae6f-metrics-certs\") pod \"maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc\" (UID: \"aa54e18d-d0ff-4c61-8377-4891f35aae6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" Sep 08 17:16:28.802579 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:28.802499 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa54e18d-d0ff-4c61-8377-4891f35aae6f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc\" (UID: \"aa54e18d-d0ff-4c61-8377-4891f35aae6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" Sep 08 17:16:28.802642 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:28.802609 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxqxz\" (UniqueName: \"kubernetes.io/projected/aa54e18d-d0ff-4c61-8377-4891f35aae6f-kube-api-access-gxqxz\") pod \"maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc\" (UID: \"aa54e18d-d0ff-4c61-8377-4891f35aae6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" Sep 08 17:16:28.802706 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:28.802689 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa54e18d-d0ff-4c61-8377-4891f35aae6f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc\" (UID: \"aa54e18d-d0ff-4c61-8377-4891f35aae6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" Sep 08 17:16:28.809139 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:28.809110 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:16:28.903636 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:28.903592 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxqxz\" (UniqueName: \"kubernetes.io/projected/aa54e18d-d0ff-4c61-8377-4891f35aae6f-kube-api-access-gxqxz\") pod \"maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc\" (UID: \"aa54e18d-d0ff-4c61-8377-4891f35aae6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" Sep 08 17:16:28.903820 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:28.903646 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa54e18d-d0ff-4c61-8377-4891f35aae6f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc\" (UID: \"aa54e18d-d0ff-4c61-8377-4891f35aae6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" Sep 08 17:16:28.903820 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:28.903712 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa54e18d-d0ff-4c61-8377-4891f35aae6f-metrics-certs\") pod \"maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc\" (UID: \"aa54e18d-d0ff-4c61-8377-4891f35aae6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" Sep 08 17:16:28.903820 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:28.903773 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa54e18d-d0ff-4c61-8377-4891f35aae6f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc\" (UID: \"aa54e18d-d0ff-4c61-8377-4891f35aae6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" Sep 08 17:16:28.903931 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:16:28.903872 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-serving-cert: secret "maas-api-e2e-ipp-b-a78a1e-serving-cert" not found Sep 08 17:16:28.903974 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:16:28.903939 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa54e18d-d0ff-4c61-8377-4891f35aae6f-maas-api-tls podName:aa54e18d-d0ff-4c61-8377-4891f35aae6f nodeName:}" failed. No retries permitted until 2026-09-08 17:16:29.40391943 +0000 UTC m=+1552.140750528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa54e18d-d0ff-4c61-8377-4891f35aae6f-maas-api-tls") pod "maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" (UID: "aa54e18d-d0ff-4c61-8377-4891f35aae6f") : secret "maas-api-e2e-ipp-b-a78a1e-serving-cert" not found Sep 08 17:16:28.904383 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:28.904354 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa54e18d-d0ff-4c61-8377-4891f35aae6f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc\" (UID: \"aa54e18d-d0ff-4c61-8377-4891f35aae6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" Sep 08 17:16:28.906257 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:28.906238 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa54e18d-d0ff-4c61-8377-4891f35aae6f-metrics-certs\") pod \"maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc\" (UID: \"aa54e18d-d0ff-4c61-8377-4891f35aae6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" Sep 08 17:16:28.912699 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:28.912616 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxqxz\" (UniqueName: \"kubernetes.io/projected/aa54e18d-d0ff-4c61-8377-4891f35aae6f-kube-api-access-gxqxz\") pod \"maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc\" (UID: \"aa54e18d-d0ff-4c61-8377-4891f35aae6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" Sep 08 17:16:28.966829 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:28.966790 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:16:29.188043 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:29.188011 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:16:29.283405 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:29.283369 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:16:29.409336 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:29.409302 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa54e18d-d0ff-4c61-8377-4891f35aae6f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc\" (UID: \"aa54e18d-d0ff-4c61-8377-4891f35aae6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" Sep 08 17:16:29.411810 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:29.411787 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa54e18d-d0ff-4c61-8377-4891f35aae6f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc\" (UID: \"aa54e18d-d0ff-4c61-8377-4891f35aae6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" Sep 08 17:16:29.575985 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:29.575890 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" Sep 08 17:16:29.851830 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:29.851789 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc"] Sep 08 17:16:29.852293 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:16:29.852270 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa54e18d_d0ff_4c61_8377_4891f35aae6f.slice/crio-bc1bb61179517a45aa863f89145e1a15c47c49b5f52d7700ef92dff68cf89441 WatchSource:0}: Error finding container bc1bb61179517a45aa863f89145e1a15c47c49b5f52d7700ef92dff68cf89441: Status 404 returned error can't find the container with id bc1bb61179517a45aa863f89145e1a15c47c49b5f52d7700ef92dff68cf89441 Sep 08 17:16:30.771240 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:30.771197 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" event={"ID":"aa54e18d-d0ff-4c61-8377-4891f35aae6f","Type":"ContainerStarted","Data":"c65d33a7a5537c21f7fd203e79bbdb3b7836d94bb95f263024290d9878e61a86"} Sep 08 17:16:30.771454 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:30.771260 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" event={"ID":"aa54e18d-d0ff-4c61-8377-4891f35aae6f","Type":"ContainerStarted","Data":"bc1bb61179517a45aa863f89145e1a15c47c49b5f52d7700ef92dff68cf89441"} Sep 08 17:16:30.771454 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:30.771380 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" Sep 08 17:16:30.789725 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:30.789676 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" podStartSLOduration=2.491125954 podStartE2EDuration="2.789659761s" totalImagesPullingTime="298.533807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:16:30.789636034 +0000 UTC m=+1553.526467152" watchObservedRunningTime="2026-09-08 17:16:30.789659761 +0000 UTC m=+1553.526490879" Sep 08 17:16:36.781773 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:36.781739 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" Sep 08 17:16:47.078705 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:47.078668 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:16:52.502091 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:52.502057 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:16:57.493656 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:16:57.493623 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-vrwnc_f5cd24d0-2d8f-4639-9fac-d4ae2ba1676d/payload-processing/0.log" Sep 08 17:17:11.009264 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:11.009217 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:17:35.481418 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:35.481336 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-vrwnc_f5cd24d0-2d8f-4639-9fac-d4ae2ba1676d/payload-processing/0.log" Sep 08 17:17:35.896193 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:35.896105 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:17:36.226959 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:36.226930 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:17:36.388193 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:36.388161 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:17:39.024111 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:39.024077 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:17:39.118700 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:39.118664 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:17:40.143596 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:40.143558 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6"] Sep 08 17:17:40.147919 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:40.147893 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" Sep 08 17:17:40.150699 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:40.150672 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-05b1b1-serving-cert\"" Sep 08 17:17:40.161252 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:40.161228 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6"] Sep 08 17:17:40.192966 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:40.192938 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:17:40.245185 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:40.245156 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a7bbc42-0ebf-4bac-87ae-78026328d146-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6\" (UID: \"7a7bbc42-0ebf-4bac-87ae-78026328d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" Sep 08 17:17:40.245357 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:40.245203 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd5qk\" (UniqueName: \"kubernetes.io/projected/7a7bbc42-0ebf-4bac-87ae-78026328d146-kube-api-access-hd5qk\") pod \"maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6\" (UID: \"7a7bbc42-0ebf-4bac-87ae-78026328d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" Sep 08 17:17:40.245406 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:40.245371 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a7bbc42-0ebf-4bac-87ae-78026328d146-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6\" (UID: \"7a7bbc42-0ebf-4bac-87ae-78026328d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" Sep 08 17:17:40.245521 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:40.245501 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7bbc42-0ebf-4bac-87ae-78026328d146-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6\" (UID: \"7a7bbc42-0ebf-4bac-87ae-78026328d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" Sep 08 17:17:40.300307 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:40.300274 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:17:40.346472 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:40.346415 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7bbc42-0ebf-4bac-87ae-78026328d146-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6\" (UID: \"7a7bbc42-0ebf-4bac-87ae-78026328d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" Sep 08 17:17:40.346598 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:40.346509 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a7bbc42-0ebf-4bac-87ae-78026328d146-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6\" (UID: \"7a7bbc42-0ebf-4bac-87ae-78026328d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" Sep 08 17:17:40.346598 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:40.346540 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd5qk\" (UniqueName: \"kubernetes.io/projected/7a7bbc42-0ebf-4bac-87ae-78026328d146-kube-api-access-hd5qk\") pod \"maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6\" (UID: \"7a7bbc42-0ebf-4bac-87ae-78026328d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" Sep 08 17:17:40.346681 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:40.346641 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a7bbc42-0ebf-4bac-87ae-78026328d146-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6\" (UID: \"7a7bbc42-0ebf-4bac-87ae-78026328d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" Sep 08 17:17:40.347281 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:40.347254 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7bbc42-0ebf-4bac-87ae-78026328d146-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6\" (UID: \"7a7bbc42-0ebf-4bac-87ae-78026328d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" Sep 08 17:17:40.348986 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:40.348964 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a7bbc42-0ebf-4bac-87ae-78026328d146-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6\" (UID: \"7a7bbc42-0ebf-4bac-87ae-78026328d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" Sep 08 17:17:40.349062 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:40.349007 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a7bbc42-0ebf-4bac-87ae-78026328d146-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6\" (UID: \"7a7bbc42-0ebf-4bac-87ae-78026328d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" Sep 08 17:17:40.355366 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:40.355340 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd5qk\" (UniqueName: \"kubernetes.io/projected/7a7bbc42-0ebf-4bac-87ae-78026328d146-kube-api-access-hd5qk\") pod \"maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6\" (UID: \"7a7bbc42-0ebf-4bac-87ae-78026328d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" Sep 08 17:17:40.460014 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:40.459985 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" Sep 08 17:17:40.712245 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:40.712159 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:17:40.744979 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:40.744935 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6"] Sep 08 17:17:40.751629 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:17:40.747293 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7bbc42_0ebf_4bac_87ae_78026328d146.slice/crio-2b607850a8573b119b75eabf411ff2e9f3d748133a18cfb4ab06faa95d18f39b WatchSource:0}: Error finding container 2b607850a8573b119b75eabf411ff2e9f3d748133a18cfb4ab06faa95d18f39b: Status 404 returned error can't find the container with id 2b607850a8573b119b75eabf411ff2e9f3d748133a18cfb4ab06faa95d18f39b Sep 08 17:17:40.813279 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:40.813235 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:17:41.189689 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:41.189654 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" event={"ID":"7a7bbc42-0ebf-4bac-87ae-78026328d146","Type":"ContainerStarted","Data":"2b607850a8573b119b75eabf411ff2e9f3d748133a18cfb4ab06faa95d18f39b"} Sep 08 17:17:42.197402 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:42.197364 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" event={"ID":"7a7bbc42-0ebf-4bac-87ae-78026328d146","Type":"ContainerStarted","Data":"834c370537cc6fc3f38f2b9ef669fffbc5087620a3658e6b2176eaca76503ada"} Sep 08 17:17:42.197871 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:42.197468 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" Sep 08 17:17:42.215987 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:42.215940 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" podStartSLOduration=1.597201104 podStartE2EDuration="2.215924069s" totalImagesPullingTime="618.722965ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:17:42.214275799 +0000 UTC m=+1624.951106939" watchObservedRunningTime="2026-09-08 17:17:42.215924069 +0000 UTC m=+1624.952755220" Sep 08 17:17:48.208869 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:17:48.208832 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" Sep 08 17:18:03.944178 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:03.944143 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:18:10.915875 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:10.915839 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6"] Sep 08 17:18:10.916373 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:10.916099 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" podUID="7a7bbc42-0ebf-4bac-87ae-78026328d146" containerName="maas-api" containerID="cri-o://834c370537cc6fc3f38f2b9ef669fffbc5087620a3658e6b2176eaca76503ada" gracePeriod=30 Sep 08 17:18:10.953475 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:10.953434 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:18:11.052534 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.052501 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:18:11.154305 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.154271 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:18:11.175005 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.174985 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" Sep 08 17:18:11.215181 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.215149 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hd5qk\" (UniqueName: \"kubernetes.io/projected/7a7bbc42-0ebf-4bac-87ae-78026328d146-kube-api-access-hd5qk\") pod \"7a7bbc42-0ebf-4bac-87ae-78026328d146\" (UID: \"7a7bbc42-0ebf-4bac-87ae-78026328d146\") " Sep 08 17:18:11.215350 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.215196 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a7bbc42-0ebf-4bac-87ae-78026328d146-metrics-certs\") pod \"7a7bbc42-0ebf-4bac-87ae-78026328d146\" (UID: \"7a7bbc42-0ebf-4bac-87ae-78026328d146\") " Sep 08 17:18:11.215350 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.215225 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a7bbc42-0ebf-4bac-87ae-78026328d146-maas-api-tls\") pod \"7a7bbc42-0ebf-4bac-87ae-78026328d146\" (UID: \"7a7bbc42-0ebf-4bac-87ae-78026328d146\") " Sep 08 17:18:11.215350 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.215249 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7bbc42-0ebf-4bac-87ae-78026328d146-openshift-service-ca\") pod \"7a7bbc42-0ebf-4bac-87ae-78026328d146\" (UID: \"7a7bbc42-0ebf-4bac-87ae-78026328d146\") " Sep 08 17:18:11.215699 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.215672 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7bbc42-0ebf-4bac-87ae-78026328d146-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7a7bbc42-0ebf-4bac-87ae-78026328d146" (UID: "7a7bbc42-0ebf-4bac-87ae-78026328d146"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:18:11.217306 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.217279 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7bbc42-0ebf-4bac-87ae-78026328d146-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7a7bbc42-0ebf-4bac-87ae-78026328d146" (UID: "7a7bbc42-0ebf-4bac-87ae-78026328d146"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:18:11.217411 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.217356 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7bbc42-0ebf-4bac-87ae-78026328d146-kube-api-access-hd5qk" (OuterVolumeSpecName: "kube-api-access-hd5qk") pod "7a7bbc42-0ebf-4bac-87ae-78026328d146" (UID: "7a7bbc42-0ebf-4bac-87ae-78026328d146"). InnerVolumeSpecName "kube-api-access-hd5qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:18:11.217411 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.217391 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7bbc42-0ebf-4bac-87ae-78026328d146-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7a7bbc42-0ebf-4bac-87ae-78026328d146" (UID: "7a7bbc42-0ebf-4bac-87ae-78026328d146"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:18:11.252181 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.252156 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:18:11.316376 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.316346 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd5qk\" (UniqueName: \"kubernetes.io/projected/7a7bbc42-0ebf-4bac-87ae-78026328d146-kube-api-access-hd5qk\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:18:11.316376 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.316374 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a7bbc42-0ebf-4bac-87ae-78026328d146-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:18:11.316608 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.316385 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a7bbc42-0ebf-4bac-87ae-78026328d146-maas-api-tls\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:18:11.316608 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.316394 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7bbc42-0ebf-4bac-87ae-78026328d146-openshift-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:18:11.370579 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.370546 2565 generic.go:356] "Generic (PLEG): container finished" podID="7a7bbc42-0ebf-4bac-87ae-78026328d146" containerID="834c370537cc6fc3f38f2b9ef669fffbc5087620a3658e6b2176eaca76503ada" exitCode=0 Sep 08 17:18:11.370679 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.370588 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" Sep 08 17:18:11.370679 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.370629 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" event={"ID":"7a7bbc42-0ebf-4bac-87ae-78026328d146","Type":"ContainerDied","Data":"834c370537cc6fc3f38f2b9ef669fffbc5087620a3658e6b2176eaca76503ada"} Sep 08 17:18:11.370767 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.370684 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6" event={"ID":"7a7bbc42-0ebf-4bac-87ae-78026328d146","Type":"ContainerDied","Data":"2b607850a8573b119b75eabf411ff2e9f3d748133a18cfb4ab06faa95d18f39b"} Sep 08 17:18:11.370767 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.370711 2565 scope.go:117] "RemoveContainer" containerID="834c370537cc6fc3f38f2b9ef669fffbc5087620a3658e6b2176eaca76503ada" Sep 08 17:18:11.380950 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.380931 2565 scope.go:117] "RemoveContainer" containerID="834c370537cc6fc3f38f2b9ef669fffbc5087620a3658e6b2176eaca76503ada" Sep 08 17:18:11.381219 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:18:11.381196 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"834c370537cc6fc3f38f2b9ef669fffbc5087620a3658e6b2176eaca76503ada\": container with ID starting with 834c370537cc6fc3f38f2b9ef669fffbc5087620a3658e6b2176eaca76503ada not found: ID does not exist" containerID="834c370537cc6fc3f38f2b9ef669fffbc5087620a3658e6b2176eaca76503ada" Sep 08 17:18:11.381263 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.381235 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834c370537cc6fc3f38f2b9ef669fffbc5087620a3658e6b2176eaca76503ada"} err="failed to get container status \"834c370537cc6fc3f38f2b9ef669fffbc5087620a3658e6b2176eaca76503ada\": rpc error: code = NotFound desc = could not find container \"834c370537cc6fc3f38f2b9ef669fffbc5087620a3658e6b2176eaca76503ada\": container with ID starting with 834c370537cc6fc3f38f2b9ef669fffbc5087620a3658e6b2176eaca76503ada not found: ID does not exist" Sep 08 17:18:11.396436 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.396405 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6"] Sep 08 17:18:11.400604 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.400584 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-05b1b1-7597fbdfb7-d75n6"] Sep 08 17:18:11.562660 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.562572 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:18:11.760947 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:11.760912 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a7bbc42-0ebf-4bac-87ae-78026328d146" path="/var/lib/kubelet/pods/7a7bbc42-0ebf-4bac-87ae-78026328d146/volumes" Sep 08 17:18:17.852978 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:17.852940 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:18:17.945002 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:17.944934 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:18:18.218068 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:18.218025 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:18:21.956581 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:21.956541 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:18:37.348945 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:37.348907 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt"] Sep 08 17:18:37.349415 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:37.349171 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" podUID="c1805bd0-8a18-44e6-975f-198b54cb6fb2" containerName="maas-api" containerID="cri-o://3a40b91490d5c11d720db491edd40730d9ed68b6c9b0368cdb76ae59bfd6186f" gracePeriod=30 Sep 08 17:18:37.533798 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:37.533766 2565 generic.go:356] "Generic (PLEG): container finished" podID="c1805bd0-8a18-44e6-975f-198b54cb6fb2" containerID="3a40b91490d5c11d720db491edd40730d9ed68b6c9b0368cdb76ae59bfd6186f" exitCode=0 Sep 08 17:18:37.533948 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:37.533825 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" event={"ID":"c1805bd0-8a18-44e6-975f-198b54cb6fb2","Type":"ContainerDied","Data":"3a40b91490d5c11d720db491edd40730d9ed68b6c9b0368cdb76ae59bfd6186f"} Sep 08 17:18:37.602605 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:37.602537 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" Sep 08 17:18:37.660668 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:37.660633 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1805bd0-8a18-44e6-975f-198b54cb6fb2-openshift-service-ca\") pod \"c1805bd0-8a18-44e6-975f-198b54cb6fb2\" (UID: \"c1805bd0-8a18-44e6-975f-198b54cb6fb2\") " Sep 08 17:18:37.660855 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:37.660702 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1805bd0-8a18-44e6-975f-198b54cb6fb2-maas-api-tls\") pod \"c1805bd0-8a18-44e6-975f-198b54cb6fb2\" (UID: \"c1805bd0-8a18-44e6-975f-198b54cb6fb2\") " Sep 08 17:18:37.660855 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:37.660746 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1805bd0-8a18-44e6-975f-198b54cb6fb2-metrics-certs\") pod \"c1805bd0-8a18-44e6-975f-198b54cb6fb2\" (UID: \"c1805bd0-8a18-44e6-975f-198b54cb6fb2\") " Sep 08 17:18:37.660855 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:37.660768 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwfx8\" (UniqueName: \"kubernetes.io/projected/c1805bd0-8a18-44e6-975f-198b54cb6fb2-kube-api-access-pwfx8\") pod \"c1805bd0-8a18-44e6-975f-198b54cb6fb2\" (UID: \"c1805bd0-8a18-44e6-975f-198b54cb6fb2\") " Sep 08 17:18:37.661041 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:37.661006 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1805bd0-8a18-44e6-975f-198b54cb6fb2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c1805bd0-8a18-44e6-975f-198b54cb6fb2" (UID: "c1805bd0-8a18-44e6-975f-198b54cb6fb2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:18:37.662850 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:37.662816 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1805bd0-8a18-44e6-975f-198b54cb6fb2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c1805bd0-8a18-44e6-975f-198b54cb6fb2" (UID: "c1805bd0-8a18-44e6-975f-198b54cb6fb2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:18:37.662962 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:37.662872 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1805bd0-8a18-44e6-975f-198b54cb6fb2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c1805bd0-8a18-44e6-975f-198b54cb6fb2" (UID: "c1805bd0-8a18-44e6-975f-198b54cb6fb2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:18:37.663026 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:37.663008 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1805bd0-8a18-44e6-975f-198b54cb6fb2-kube-api-access-pwfx8" (OuterVolumeSpecName: "kube-api-access-pwfx8") pod "c1805bd0-8a18-44e6-975f-198b54cb6fb2" (UID: "c1805bd0-8a18-44e6-975f-198b54cb6fb2"). InnerVolumeSpecName "kube-api-access-pwfx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:18:37.762308 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:37.762272 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1805bd0-8a18-44e6-975f-198b54cb6fb2-maas-api-tls\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:18:37.762308 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:37.762306 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1805bd0-8a18-44e6-975f-198b54cb6fb2-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:18:37.762562 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:37.762319 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwfx8\" (UniqueName: \"kubernetes.io/projected/c1805bd0-8a18-44e6-975f-198b54cb6fb2-kube-api-access-pwfx8\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:18:37.762562 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:37.762332 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1805bd0-8a18-44e6-975f-198b54cb6fb2-openshift-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:18:38.547326 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:38.547285 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" event={"ID":"c1805bd0-8a18-44e6-975f-198b54cb6fb2","Type":"ContainerDied","Data":"c69ba91da5e1564bb185613e879154154a93ed45cffa823aacd764c8755154b5"} Sep 08 17:18:38.547805 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:38.547329 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt" Sep 08 17:18:38.547805 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:38.547391 2565 scope.go:117] "RemoveContainer" containerID="3a40b91490d5c11d720db491edd40730d9ed68b6c9b0368cdb76ae59bfd6186f" Sep 08 17:18:38.570371 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:38.570346 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt"] Sep 08 17:18:38.574358 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:38.574338 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-978af9-679dd97f5-f46wt"] Sep 08 17:18:39.071670 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:39.071639 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:18:39.138747 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:39.138712 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:18:39.761406 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:39.761366 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1805bd0-8a18-44e6-975f-198b54cb6fb2" path="/var/lib/kubelet/pods/c1805bd0-8a18-44e6-975f-198b54cb6fb2/volumes" Sep 08 17:18:43.057158 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:43.057119 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:18:49.217321 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:49.217285 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:18:49.308731 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:49.308693 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:18:51.970307 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:18:51.970272 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:06.994906 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:06.994790 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:21.172076 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:21.172043 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:21.241490 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:21.241459 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:26.500364 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:26.500329 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:26.602060 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:26.602025 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:26.807708 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:26.807612 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:27.367272 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:27.367235 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:37.730995 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:37.730954 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc"] Sep 08 17:19:37.731775 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:37.731710 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" podUID="aa54e18d-d0ff-4c61-8377-4891f35aae6f" containerName="maas-api" containerID="cri-o://c65d33a7a5537c21f7fd203e79bbdb3b7836d94bb95f263024290d9878e61a86" gracePeriod=30 Sep 08 17:19:37.761082 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:37.761044 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:37.842185 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:37.842151 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:37.895664 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:37.895631 2565 generic.go:356] "Generic (PLEG): container finished" podID="aa54e18d-d0ff-4c61-8377-4891f35aae6f" containerID="c65d33a7a5537c21f7fd203e79bbdb3b7836d94bb95f263024290d9878e61a86" exitCode=0 Sep 08 17:19:37.895827 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:37.895687 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" event={"ID":"aa54e18d-d0ff-4c61-8377-4891f35aae6f","Type":"ContainerDied","Data":"c65d33a7a5537c21f7fd203e79bbdb3b7836d94bb95f263024290d9878e61a86"} Sep 08 17:19:37.922752 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:37.922515 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:37.991446 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:37.991382 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" Sep 08 17:19:38.013901 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:38.013786 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:38.093282 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:38.093255 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:38.102215 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:38.102191 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa54e18d-d0ff-4c61-8377-4891f35aae6f-maas-api-tls\") pod \"aa54e18d-d0ff-4c61-8377-4891f35aae6f\" (UID: \"aa54e18d-d0ff-4c61-8377-4891f35aae6f\") " Sep 08 17:19:38.102348 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:38.102246 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa54e18d-d0ff-4c61-8377-4891f35aae6f-metrics-certs\") pod \"aa54e18d-d0ff-4c61-8377-4891f35aae6f\" (UID: \"aa54e18d-d0ff-4c61-8377-4891f35aae6f\") " Sep 08 17:19:38.102348 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:38.102341 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxqxz\" (UniqueName: \"kubernetes.io/projected/aa54e18d-d0ff-4c61-8377-4891f35aae6f-kube-api-access-gxqxz\") pod \"aa54e18d-d0ff-4c61-8377-4891f35aae6f\" (UID: \"aa54e18d-d0ff-4c61-8377-4891f35aae6f\") " Sep 08 17:19:38.102536 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:38.102397 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa54e18d-d0ff-4c61-8377-4891f35aae6f-openshift-service-ca\") pod \"aa54e18d-d0ff-4c61-8377-4891f35aae6f\" (UID: \"aa54e18d-d0ff-4c61-8377-4891f35aae6f\") " Sep 08 17:19:38.102830 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:38.102800 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa54e18d-d0ff-4c61-8377-4891f35aae6f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aa54e18d-d0ff-4c61-8377-4891f35aae6f" (UID: "aa54e18d-d0ff-4c61-8377-4891f35aae6f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:19:38.104460 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:38.104416 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa54e18d-d0ff-4c61-8377-4891f35aae6f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aa54e18d-d0ff-4c61-8377-4891f35aae6f" (UID: "aa54e18d-d0ff-4c61-8377-4891f35aae6f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:19:38.104584 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:38.104564 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa54e18d-d0ff-4c61-8377-4891f35aae6f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "aa54e18d-d0ff-4c61-8377-4891f35aae6f" (UID: "aa54e18d-d0ff-4c61-8377-4891f35aae6f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:19:38.104645 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:38.104588 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa54e18d-d0ff-4c61-8377-4891f35aae6f-kube-api-access-gxqxz" (OuterVolumeSpecName: "kube-api-access-gxqxz") pod "aa54e18d-d0ff-4c61-8377-4891f35aae6f" (UID: "aa54e18d-d0ff-4c61-8377-4891f35aae6f"). InnerVolumeSpecName "kube-api-access-gxqxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:19:38.203577 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:38.203544 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa54e18d-d0ff-4c61-8377-4891f35aae6f-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:19:38.203577 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:38.203572 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxqxz\" (UniqueName: \"kubernetes.io/projected/aa54e18d-d0ff-4c61-8377-4891f35aae6f-kube-api-access-gxqxz\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:19:38.203577 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:38.203582 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa54e18d-d0ff-4c61-8377-4891f35aae6f-openshift-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:19:38.203991 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:38.203591 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa54e18d-d0ff-4c61-8377-4891f35aae6f-maas-api-tls\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:19:38.395693 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:38.395610 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:38.903743 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:38.903715 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" Sep 08 17:19:38.904150 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:38.903746 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc" event={"ID":"aa54e18d-d0ff-4c61-8377-4891f35aae6f","Type":"ContainerDied","Data":"bc1bb61179517a45aa863f89145e1a15c47c49b5f52d7700ef92dff68cf89441"} Sep 08 17:19:38.904150 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:38.903823 2565 scope.go:117] "RemoveContainer" containerID="c65d33a7a5537c21f7fd203e79bbdb3b7836d94bb95f263024290d9878e61a86" Sep 08 17:19:38.930284 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:38.930255 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc"] Sep 08 17:19:38.932950 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:38.932925 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a78a1e-7ccbbcb4ff-59zfc"] Sep 08 17:19:39.762071 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:39.762035 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa54e18d-d0ff-4c61-8377-4891f35aae6f" path="/var/lib/kubelet/pods/aa54e18d-d0ff-4c61-8377-4891f35aae6f/volumes" Sep 08 17:19:44.569119 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:44.569076 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:44.679594 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:44.679561 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:44.837619 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:44.837544 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:45.283524 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:45.283491 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:55.705892 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:55.705854 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z"] Sep 08 17:19:55.706483 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:55.706069 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" podUID="a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d" containerName="maas-api" containerID="cri-o://7066f364fde421972bf7e77d622a3473a493a3103ae506cac82e25b38402d2a0" gracePeriod=30 Sep 08 17:19:55.739059 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:55.739023 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:55.877688 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:55.877498 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:55.962919 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:55.962861 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" Sep 08 17:19:55.975809 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:55.975759 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:56.007856 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.007822 2565 generic.go:356] "Generic (PLEG): container finished" podID="a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d" containerID="7066f364fde421972bf7e77d622a3473a493a3103ae506cac82e25b38402d2a0" exitCode=0 Sep 08 17:19:56.008001 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.007867 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" Sep 08 17:19:56.008001 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.007913 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" event={"ID":"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d","Type":"ContainerDied","Data":"7066f364fde421972bf7e77d622a3473a493a3103ae506cac82e25b38402d2a0"} Sep 08 17:19:56.008001 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.007973 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z" event={"ID":"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d","Type":"ContainerDied","Data":"b97a1a4e7fdb7373320ccdfbe336ce09b71a911ddcc3175e5964db7caa9c7193"} Sep 08 17:19:56.008128 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.008006 2565 scope.go:117] "RemoveContainer" containerID="7066f364fde421972bf7e77d622a3473a493a3103ae506cac82e25b38402d2a0" Sep 08 17:19:56.017154 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.017137 2565 scope.go:117] "RemoveContainer" containerID="7066f364fde421972bf7e77d622a3473a493a3103ae506cac82e25b38402d2a0" Sep 08 17:19:56.017398 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:19:56.017378 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7066f364fde421972bf7e77d622a3473a493a3103ae506cac82e25b38402d2a0\": container with ID starting with 7066f364fde421972bf7e77d622a3473a493a3103ae506cac82e25b38402d2a0 not found: ID does not exist" containerID="7066f364fde421972bf7e77d622a3473a493a3103ae506cac82e25b38402d2a0" Sep 08 17:19:56.017489 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.017406 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7066f364fde421972bf7e77d622a3473a493a3103ae506cac82e25b38402d2a0"} err="failed to get container status \"7066f364fde421972bf7e77d622a3473a493a3103ae506cac82e25b38402d2a0\": rpc error: code = NotFound desc = could not find container \"7066f364fde421972bf7e77d622a3473a493a3103ae506cac82e25b38402d2a0\": container with ID starting with 7066f364fde421972bf7e77d622a3473a493a3103ae506cac82e25b38402d2a0 not found: ID does not exist" Sep 08 17:19:56.057281 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.057261 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nctld\" (UniqueName: \"kubernetes.io/projected/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-kube-api-access-nctld\") pod \"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d\" (UID: \"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d\") " Sep 08 17:19:56.057381 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.057297 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-maas-api-tls\") pod \"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d\" (UID: \"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d\") " Sep 08 17:19:56.057381 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.057349 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-openshift-service-ca\") pod \"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d\" (UID: \"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d\") " Sep 08 17:19:56.057508 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.057386 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-metrics-certs\") pod \"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d\" (UID: \"a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d\") " Sep 08 17:19:56.057758 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.057731 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d" (UID: "a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:19:56.059452 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.059413 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d" (UID: "a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:19:56.059540 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.059490 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-kube-api-access-nctld" (OuterVolumeSpecName: "kube-api-access-nctld") pod "a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d" (UID: "a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d"). InnerVolumeSpecName "kube-api-access-nctld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:19:56.059586 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.059547 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d" (UID: "a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:19:56.074007 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.073984 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:56.158261 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.158233 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nctld\" (UniqueName: \"kubernetes.io/projected/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-kube-api-access-nctld\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:19:56.158379 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.158266 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-maas-api-tls\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:19:56.158379 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.158283 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-openshift-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:19:56.158379 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.158297 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:19:56.335383 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.335350 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z"] Sep 08 17:19:56.337608 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.337586 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c05c9f-77cf8cffcb-lcc5z"] Sep 08 17:19:56.371225 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:56.371197 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:19:57.761075 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:57.761043 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d" path="/var/lib/kubelet/pods/a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d/volumes" Sep 08 17:19:59.856722 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:19:59.856684 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:20:02.664091 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:02.664053 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:20:02.763870 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:02.763833 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:20:02.836694 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:02.836648 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:20:03.359627 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:03.359591 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:20:13.641054 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:13.641010 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6"] Sep 08 17:20:13.641554 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:13.641287 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" podUID="0019913a-b578-414a-8e5c-d03cd746c1d0" containerName="maas-api" containerID="cri-o://5540178d0dd3793be268b09fd548118d5c3f67e60e230c42522f90da0cb6ed53" gracePeriod=30 Sep 08 17:20:13.672750 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:13.672716 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:20:13.774400 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:13.774318 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:20:13.872087 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:13.872052 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:20:13.889352 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:13.889326 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" Sep 08 17:20:13.903670 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:13.903608 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0019913a-b578-414a-8e5c-d03cd746c1d0-metrics-certs\") pod \"0019913a-b578-414a-8e5c-d03cd746c1d0\" (UID: \"0019913a-b578-414a-8e5c-d03cd746c1d0\") " Sep 08 17:20:13.903771 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:13.903670 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0019913a-b578-414a-8e5c-d03cd746c1d0-maas-api-tls\") pod \"0019913a-b578-414a-8e5c-d03cd746c1d0\" (UID: \"0019913a-b578-414a-8e5c-d03cd746c1d0\") " Sep 08 17:20:13.903771 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:13.903730 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbxnz\" (UniqueName: \"kubernetes.io/projected/0019913a-b578-414a-8e5c-d03cd746c1d0-kube-api-access-cbxnz\") pod \"0019913a-b578-414a-8e5c-d03cd746c1d0\" (UID: \"0019913a-b578-414a-8e5c-d03cd746c1d0\") " Sep 08 17:20:13.903860 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:13.903809 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0019913a-b578-414a-8e5c-d03cd746c1d0-openshift-service-ca\") pod \"0019913a-b578-414a-8e5c-d03cd746c1d0\" (UID: \"0019913a-b578-414a-8e5c-d03cd746c1d0\") " Sep 08 17:20:13.904231 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:13.904197 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0019913a-b578-414a-8e5c-d03cd746c1d0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0019913a-b578-414a-8e5c-d03cd746c1d0" (UID: "0019913a-b578-414a-8e5c-d03cd746c1d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:20:13.905815 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:13.905793 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0019913a-b578-414a-8e5c-d03cd746c1d0-kube-api-access-cbxnz" (OuterVolumeSpecName: "kube-api-access-cbxnz") pod "0019913a-b578-414a-8e5c-d03cd746c1d0" (UID: "0019913a-b578-414a-8e5c-d03cd746c1d0"). InnerVolumeSpecName "kube-api-access-cbxnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:20:13.906202 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:13.906183 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0019913a-b578-414a-8e5c-d03cd746c1d0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0019913a-b578-414a-8e5c-d03cd746c1d0" (UID: "0019913a-b578-414a-8e5c-d03cd746c1d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:20:13.906267 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:13.906208 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0019913a-b578-414a-8e5c-d03cd746c1d0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0019913a-b578-414a-8e5c-d03cd746c1d0" (UID: "0019913a-b578-414a-8e5c-d03cd746c1d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:20:13.972095 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:13.972039 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:20:14.005198 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:14.005168 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0019913a-b578-414a-8e5c-d03cd746c1d0-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:20:14.005198 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:14.005193 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0019913a-b578-414a-8e5c-d03cd746c1d0-maas-api-tls\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:20:14.005198 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:14.005204 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbxnz\" (UniqueName: \"kubernetes.io/projected/0019913a-b578-414a-8e5c-d03cd746c1d0-kube-api-access-cbxnz\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:20:14.005448 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:14.005213 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0019913a-b578-414a-8e5c-d03cd746c1d0-openshift-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:20:14.131958 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:14.131921 2565 generic.go:356] "Generic (PLEG): container finished" podID="0019913a-b578-414a-8e5c-d03cd746c1d0" containerID="5540178d0dd3793be268b09fd548118d5c3f67e60e230c42522f90da0cb6ed53" exitCode=0 Sep 08 17:20:14.132156 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:14.131986 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" Sep 08 17:20:14.132156 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:14.132005 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" event={"ID":"0019913a-b578-414a-8e5c-d03cd746c1d0","Type":"ContainerDied","Data":"5540178d0dd3793be268b09fd548118d5c3f67e60e230c42522f90da0cb6ed53"} Sep 08 17:20:14.132156 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:14.132098 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6" event={"ID":"0019913a-b578-414a-8e5c-d03cd746c1d0","Type":"ContainerDied","Data":"5c78c2b32f8ac9019449527c8ae3a84a1d0a2a09a985a7262d5dca935f99e128"} Sep 08 17:20:14.132156 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:14.132129 2565 scope.go:117] "RemoveContainer" containerID="5540178d0dd3793be268b09fd548118d5c3f67e60e230c42522f90da0cb6ed53" Sep 08 17:20:14.142150 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:14.142132 2565 scope.go:117] "RemoveContainer" containerID="5540178d0dd3793be268b09fd548118d5c3f67e60e230c42522f90da0cb6ed53" Sep 08 17:20:14.142404 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:20:14.142387 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5540178d0dd3793be268b09fd548118d5c3f67e60e230c42522f90da0cb6ed53\": container with ID starting with 5540178d0dd3793be268b09fd548118d5c3f67e60e230c42522f90da0cb6ed53 not found: ID does not exist" containerID="5540178d0dd3793be268b09fd548118d5c3f67e60e230c42522f90da0cb6ed53" Sep 08 17:20:14.142482 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:14.142409 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5540178d0dd3793be268b09fd548118d5c3f67e60e230c42522f90da0cb6ed53"} err="failed to get container status \"5540178d0dd3793be268b09fd548118d5c3f67e60e230c42522f90da0cb6ed53\": rpc error: code = NotFound desc = could not find container \"5540178d0dd3793be268b09fd548118d5c3f67e60e230c42522f90da0cb6ed53\": container with ID starting with 5540178d0dd3793be268b09fd548118d5c3f67e60e230c42522f90da0cb6ed53 not found: ID does not exist" Sep 08 17:20:14.155627 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:14.155568 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6"] Sep 08 17:20:14.157879 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:14.157856 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-338600-78b768495f-7hzr6"] Sep 08 17:20:14.273220 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:14.273186 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:20:15.760111 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:15.760080 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0019913a-b578-414a-8e5c-d03cd746c1d0" path="/var/lib/kubelet/pods/0019913a-b578-414a-8e5c-d03cd746c1d0/volumes" Sep 08 17:20:20.535539 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:20.535506 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:20:20.635983 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:20.635948 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:20:20.821320 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:20.821239 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:20:23.491960 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:23.491915 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75d5cb9587-vt7ng"] Sep 08 17:20:23.492394 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:23.492143 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-75d5cb9587-vt7ng" podUID="ccb45355-528f-453d-ad2d-210d4e8963ab" containerName="manager" containerID="cri-o://d4abcef5a7053b1ba59a4d2126741ab31f821a343007e72f575604ce5f9ab7ee" gracePeriod=30 Sep 08 17:20:23.732681 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:23.732655 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75d5cb9587-vt7ng" Sep 08 17:20:23.786179 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:23.786106 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ccb45355-528f-453d-ad2d-210d4e8963ab-webhook-cert\") pod \"ccb45355-528f-453d-ad2d-210d4e8963ab\" (UID: \"ccb45355-528f-453d-ad2d-210d4e8963ab\") " Sep 08 17:20:23.786179 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:23.786166 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n95x4\" (UniqueName: \"kubernetes.io/projected/ccb45355-528f-453d-ad2d-210d4e8963ab-kube-api-access-n95x4\") pod \"ccb45355-528f-453d-ad2d-210d4e8963ab\" (UID: \"ccb45355-528f-453d-ad2d-210d4e8963ab\") " Sep 08 17:20:23.786397 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:23.786196 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccb45355-528f-453d-ad2d-210d4e8963ab-metrics-certs\") pod \"ccb45355-528f-453d-ad2d-210d4e8963ab\" (UID: \"ccb45355-528f-453d-ad2d-210d4e8963ab\") " Sep 08 17:20:23.788466 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:23.788415 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb45355-528f-453d-ad2d-210d4e8963ab-kube-api-access-n95x4" (OuterVolumeSpecName: "kube-api-access-n95x4") pod "ccb45355-528f-453d-ad2d-210d4e8963ab" (UID: "ccb45355-528f-453d-ad2d-210d4e8963ab"). InnerVolumeSpecName "kube-api-access-n95x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:20:23.788694 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:23.788673 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb45355-528f-453d-ad2d-210d4e8963ab-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ccb45355-528f-453d-ad2d-210d4e8963ab" (UID: "ccb45355-528f-453d-ad2d-210d4e8963ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:20:23.788787 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:23.788710 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb45355-528f-453d-ad2d-210d4e8963ab-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ccb45355-528f-453d-ad2d-210d4e8963ab" (UID: "ccb45355-528f-453d-ad2d-210d4e8963ab"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:20:23.887785 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:23.887755 2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ccb45355-528f-453d-ad2d-210d4e8963ab-webhook-cert\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:20:23.887785 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:23.887784 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n95x4\" (UniqueName: \"kubernetes.io/projected/ccb45355-528f-453d-ad2d-210d4e8963ab-kube-api-access-n95x4\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:20:23.887953 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:23.887794 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccb45355-528f-453d-ad2d-210d4e8963ab-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:20:24.194279 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:24.194245 2565 generic.go:356] "Generic (PLEG): container finished" podID="ccb45355-528f-453d-ad2d-210d4e8963ab" containerID="d4abcef5a7053b1ba59a4d2126741ab31f821a343007e72f575604ce5f9ab7ee" exitCode=0 Sep 08 17:20:24.194480 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:24.194289 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75d5cb9587-vt7ng" Sep 08 17:20:24.194480 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:24.194323 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75d5cb9587-vt7ng" event={"ID":"ccb45355-528f-453d-ad2d-210d4e8963ab","Type":"ContainerDied","Data":"d4abcef5a7053b1ba59a4d2126741ab31f821a343007e72f575604ce5f9ab7ee"} Sep 08 17:20:24.194480 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:24.194373 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75d5cb9587-vt7ng" event={"ID":"ccb45355-528f-453d-ad2d-210d4e8963ab","Type":"ContainerDied","Data":"0233a1e08faca43a79eb7e5ebfa1d174da134ee9ea71beb72db5ccbf8aafcfc5"} Sep 08 17:20:24.194480 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:24.194397 2565 scope.go:117] "RemoveContainer" containerID="d4abcef5a7053b1ba59a4d2126741ab31f821a343007e72f575604ce5f9ab7ee" Sep 08 17:20:24.204864 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:24.204847 2565 scope.go:117] "RemoveContainer" containerID="d4abcef5a7053b1ba59a4d2126741ab31f821a343007e72f575604ce5f9ab7ee" Sep 08 17:20:24.205128 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:20:24.205108 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4abcef5a7053b1ba59a4d2126741ab31f821a343007e72f575604ce5f9ab7ee\": container with ID starting with d4abcef5a7053b1ba59a4d2126741ab31f821a343007e72f575604ce5f9ab7ee not found: ID does not exist" containerID="d4abcef5a7053b1ba59a4d2126741ab31f821a343007e72f575604ce5f9ab7ee" Sep 08 17:20:24.205171 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:24.205136 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4abcef5a7053b1ba59a4d2126741ab31f821a343007e72f575604ce5f9ab7ee"} err="failed to get container status \"d4abcef5a7053b1ba59a4d2126741ab31f821a343007e72f575604ce5f9ab7ee\": rpc error: code = NotFound desc = could not find container \"d4abcef5a7053b1ba59a4d2126741ab31f821a343007e72f575604ce5f9ab7ee\": container with ID starting with d4abcef5a7053b1ba59a4d2126741ab31f821a343007e72f575604ce5f9ab7ee not found: ID does not exist" Sep 08 17:20:24.219906 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:24.219881 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75d5cb9587-vt7ng"] Sep 08 17:20:24.225948 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:24.225927 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75d5cb9587-vt7ng"] Sep 08 17:20:25.760675 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:25.760641 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccb45355-528f-453d-ad2d-210d4e8963ab" path="/var/lib/kubelet/pods/ccb45355-528f-453d-ad2d-210d4e8963ab/volumes" Sep 08 17:20:31.623486 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:31.623391 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:20:37.902089 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:37.902061 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/2.log" Sep 08 17:20:37.911350 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:37.911322 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/2.log" Sep 08 17:20:37.911962 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:37.911938 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xxnl_bf53ac63-f864-402b-bf12-45b289e7e13d/ovn-acl-logging/0.log" Sep 08 17:20:37.923892 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:20:37.923872 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xxnl_bf53ac63-f864-402b-bf12-45b289e7e13d/ovn-acl-logging/0.log" Sep 08 17:21:25.051668 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.051627 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75d5cb9587-qb5pj"] Sep 08 17:21:25.052128 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.052013 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d" containerName="maas-api" Sep 08 17:21:25.052128 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.052024 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d" containerName="maas-api" Sep 08 17:21:25.052128 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.052031 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb45355-528f-453d-ad2d-210d4e8963ab" containerName="manager" Sep 08 17:21:25.052128 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.052037 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb45355-528f-453d-ad2d-210d4e8963ab" containerName="manager" Sep 08 17:21:25.052128 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.052046 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa54e18d-d0ff-4c61-8377-4891f35aae6f" containerName="maas-api" Sep 08 17:21:25.052128 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.052051 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa54e18d-d0ff-4c61-8377-4891f35aae6f" containerName="maas-api" Sep 08 17:21:25.052128 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.052060 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1805bd0-8a18-44e6-975f-198b54cb6fb2" containerName="maas-api" Sep 08 17:21:25.052128 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.052065 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1805bd0-8a18-44e6-975f-198b54cb6fb2" containerName="maas-api" Sep 08 17:21:25.052128 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.052076 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0019913a-b578-414a-8e5c-d03cd746c1d0" containerName="maas-api" Sep 08 17:21:25.052128 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.052083 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0019913a-b578-414a-8e5c-d03cd746c1d0" containerName="maas-api" Sep 08 17:21:25.052128 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.052090 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7bbc42-0ebf-4bac-87ae-78026328d146" containerName="maas-api" Sep 08 17:21:25.052128 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.052095 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7bbc42-0ebf-4bac-87ae-78026328d146" containerName="maas-api" Sep 08 17:21:25.052584 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.052154 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1409c50-bcbc-48c7-b7ab-dba7bed9ab0d" containerName="maas-api" Sep 08 17:21:25.052584 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.052161 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1805bd0-8a18-44e6-975f-198b54cb6fb2" containerName="maas-api" Sep 08 17:21:25.052584 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.052168 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0019913a-b578-414a-8e5c-d03cd746c1d0" containerName="maas-api" Sep 08 17:21:25.052584 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.052173 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa54e18d-d0ff-4c61-8377-4891f35aae6f" containerName="maas-api" Sep 08 17:21:25.052584 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.052179 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a7bbc42-0ebf-4bac-87ae-78026328d146" containerName="maas-api" Sep 08 17:21:25.052584 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.052185 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb45355-528f-453d-ad2d-210d4e8963ab" containerName="manager" Sep 08 17:21:25.055314 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.055297 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75d5cb9587-qb5pj" Sep 08 17:21:25.058673 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.058648 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 08 17:21:25.058960 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.058938 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 08 17:21:25.059025 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.058947 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j7x8d\"" Sep 08 17:21:25.059025 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.058983 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 08 17:21:25.071266 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.071237 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75d5cb9587-qb5pj"] Sep 08 17:21:25.102614 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.102573 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f458df5-6b01-4d03-a598-e77744281cc9-webhook-cert\") pod \"maas-controller-75d5cb9587-qb5pj\" (UID: \"4f458df5-6b01-4d03-a598-e77744281cc9\") " pod="opendatahub/maas-controller-75d5cb9587-qb5pj" Sep 08 17:21:25.102793 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.102638 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f458df5-6b01-4d03-a598-e77744281cc9-metrics-certs\") pod \"maas-controller-75d5cb9587-qb5pj\" (UID: \"4f458df5-6b01-4d03-a598-e77744281cc9\") " pod="opendatahub/maas-controller-75d5cb9587-qb5pj" Sep 08 17:21:25.102793 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.102698 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn9l2\" (UniqueName: \"kubernetes.io/projected/4f458df5-6b01-4d03-a598-e77744281cc9-kube-api-access-xn9l2\") pod \"maas-controller-75d5cb9587-qb5pj\" (UID: \"4f458df5-6b01-4d03-a598-e77744281cc9\") " pod="opendatahub/maas-controller-75d5cb9587-qb5pj" Sep 08 17:21:25.203158 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.203116 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn9l2\" (UniqueName: \"kubernetes.io/projected/4f458df5-6b01-4d03-a598-e77744281cc9-kube-api-access-xn9l2\") pod \"maas-controller-75d5cb9587-qb5pj\" (UID: \"4f458df5-6b01-4d03-a598-e77744281cc9\") " pod="opendatahub/maas-controller-75d5cb9587-qb5pj" Sep 08 17:21:25.203346 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.203178 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f458df5-6b01-4d03-a598-e77744281cc9-webhook-cert\") pod \"maas-controller-75d5cb9587-qb5pj\" (UID: \"4f458df5-6b01-4d03-a598-e77744281cc9\") " pod="opendatahub/maas-controller-75d5cb9587-qb5pj" Sep 08 17:21:25.203346 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.203223 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f458df5-6b01-4d03-a598-e77744281cc9-metrics-certs\") pod \"maas-controller-75d5cb9587-qb5pj\" (UID: \"4f458df5-6b01-4d03-a598-e77744281cc9\") " pod="opendatahub/maas-controller-75d5cb9587-qb5pj" Sep 08 17:21:25.205777 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.205744 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f458df5-6b01-4d03-a598-e77744281cc9-webhook-cert\") pod \"maas-controller-75d5cb9587-qb5pj\" (UID: \"4f458df5-6b01-4d03-a598-e77744281cc9\") " pod="opendatahub/maas-controller-75d5cb9587-qb5pj" Sep 08 17:21:25.205777 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.205771 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f458df5-6b01-4d03-a598-e77744281cc9-metrics-certs\") pod \"maas-controller-75d5cb9587-qb5pj\" (UID: \"4f458df5-6b01-4d03-a598-e77744281cc9\") " pod="opendatahub/maas-controller-75d5cb9587-qb5pj" Sep 08 17:21:25.212414 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.212385 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn9l2\" (UniqueName: \"kubernetes.io/projected/4f458df5-6b01-4d03-a598-e77744281cc9-kube-api-access-xn9l2\") pod \"maas-controller-75d5cb9587-qb5pj\" (UID: \"4f458df5-6b01-4d03-a598-e77744281cc9\") " pod="opendatahub/maas-controller-75d5cb9587-qb5pj" Sep 08 17:21:25.366148 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.366041 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75d5cb9587-qb5pj" Sep 08 17:21:25.648383 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.648350 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75d5cb9587-qb5pj"] Sep 08 17:21:25.650154 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:21:25.650125 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f458df5_6b01_4d03_a598_e77744281cc9.slice/crio-a32662be7f0884723e1c630b7eac39334feaf448245941865bb825e809c98213 WatchSource:0}: Error finding container a32662be7f0884723e1c630b7eac39334feaf448245941865bb825e809c98213: Status 404 returned error can't find the container with id a32662be7f0884723e1c630b7eac39334feaf448245941865bb825e809c98213 Sep 08 17:21:25.651453 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:25.651414 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 17:21:26.541060 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:26.541020 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75d5cb9587-qb5pj" event={"ID":"4f458df5-6b01-4d03-a598-e77744281cc9","Type":"ContainerStarted","Data":"a32662be7f0884723e1c630b7eac39334feaf448245941865bb825e809c98213"} Sep 08 17:21:28.554096 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:28.554002 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75d5cb9587-qb5pj" event={"ID":"4f458df5-6b01-4d03-a598-e77744281cc9","Type":"ContainerStarted","Data":"71c103c8ceb2a97dec7c1cd5ebbfca8e6b90553394021e73854a018ceb59c9cc"} Sep 08 17:21:28.554613 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:28.554109 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75d5cb9587-qb5pj" Sep 08 17:21:39.562073 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:39.562040 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75d5cb9587-qb5pj" Sep 08 17:21:39.586987 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:39.586934 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-75d5cb9587-qb5pj" podStartSLOduration=11.996116309 podStartE2EDuration="14.586916459s" totalImagesPullingTime="2.59080015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:21:28.583745965 +0000 UTC m=+1851.320577080" watchObservedRunningTime="2026-09-08 17:21:39.586916459 +0000 UTC m=+1862.323747604" Sep 08 17:21:46.738383 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:46.738350 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:21:48.773199 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:48.773165 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:21:48.877416 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:48.877374 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:21:48.944562 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:48.944525 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75d5cb9587-qb5pj"] Sep 08 17:21:48.944840 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:48.944797 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-75d5cb9587-qb5pj" podUID="4f458df5-6b01-4d03-a598-e77744281cc9" containerName="manager" containerID="cri-o://71c103c8ceb2a97dec7c1cd5ebbfca8e6b90553394021e73854a018ceb59c9cc" gracePeriod=30 Sep 08 17:21:49.191781 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:49.191752 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75d5cb9587-qb5pj" Sep 08 17:21:49.324136 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:49.324039 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f458df5-6b01-4d03-a598-e77744281cc9-webhook-cert\") pod \"4f458df5-6b01-4d03-a598-e77744281cc9\" (UID: \"4f458df5-6b01-4d03-a598-e77744281cc9\") " Sep 08 17:21:49.324136 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:49.324092 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f458df5-6b01-4d03-a598-e77744281cc9-metrics-certs\") pod \"4f458df5-6b01-4d03-a598-e77744281cc9\" (UID: \"4f458df5-6b01-4d03-a598-e77744281cc9\") " Sep 08 17:21:49.324386 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:49.324139 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xn9l2\" (UniqueName: \"kubernetes.io/projected/4f458df5-6b01-4d03-a598-e77744281cc9-kube-api-access-xn9l2\") pod \"4f458df5-6b01-4d03-a598-e77744281cc9\" (UID: \"4f458df5-6b01-4d03-a598-e77744281cc9\") " Sep 08 17:21:49.326300 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:49.326267 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f458df5-6b01-4d03-a598-e77744281cc9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4f458df5-6b01-4d03-a598-e77744281cc9" (UID: "4f458df5-6b01-4d03-a598-e77744281cc9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:21:49.326300 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:49.326284 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f458df5-6b01-4d03-a598-e77744281cc9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4f458df5-6b01-4d03-a598-e77744281cc9" (UID: "4f458df5-6b01-4d03-a598-e77744281cc9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:21:49.326300 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:49.326263 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f458df5-6b01-4d03-a598-e77744281cc9-kube-api-access-xn9l2" (OuterVolumeSpecName: "kube-api-access-xn9l2") pod "4f458df5-6b01-4d03-a598-e77744281cc9" (UID: "4f458df5-6b01-4d03-a598-e77744281cc9"). InnerVolumeSpecName "kube-api-access-xn9l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:21:49.425353 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:49.425315 2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f458df5-6b01-4d03-a598-e77744281cc9-webhook-cert\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:21:49.425353 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:49.425345 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f458df5-6b01-4d03-a598-e77744281cc9-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:21:49.425353 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:49.425355 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn9l2\" (UniqueName: \"kubernetes.io/projected/4f458df5-6b01-4d03-a598-e77744281cc9-kube-api-access-xn9l2\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:21:49.679773 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:49.679735 2565 generic.go:356] "Generic (PLEG): container finished" podID="4f458df5-6b01-4d03-a598-e77744281cc9" containerID="71c103c8ceb2a97dec7c1cd5ebbfca8e6b90553394021e73854a018ceb59c9cc" exitCode=0 Sep 08 17:21:49.679962 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:49.679785 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75d5cb9587-qb5pj" Sep 08 17:21:49.679962 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:49.679826 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75d5cb9587-qb5pj" event={"ID":"4f458df5-6b01-4d03-a598-e77744281cc9","Type":"ContainerDied","Data":"71c103c8ceb2a97dec7c1cd5ebbfca8e6b90553394021e73854a018ceb59c9cc"} Sep 08 17:21:49.679962 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:49.679881 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75d5cb9587-qb5pj" event={"ID":"4f458df5-6b01-4d03-a598-e77744281cc9","Type":"ContainerDied","Data":"a32662be7f0884723e1c630b7eac39334feaf448245941865bb825e809c98213"} Sep 08 17:21:49.679962 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:49.679911 2565 scope.go:117] "RemoveContainer" containerID="71c103c8ceb2a97dec7c1cd5ebbfca8e6b90553394021e73854a018ceb59c9cc" Sep 08 17:21:49.689989 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:49.689970 2565 scope.go:117] "RemoveContainer" containerID="71c103c8ceb2a97dec7c1cd5ebbfca8e6b90553394021e73854a018ceb59c9cc" Sep 08 17:21:49.690293 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:21:49.690274 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c103c8ceb2a97dec7c1cd5ebbfca8e6b90553394021e73854a018ceb59c9cc\": container with ID starting with 71c103c8ceb2a97dec7c1cd5ebbfca8e6b90553394021e73854a018ceb59c9cc not found: ID does not exist" containerID="71c103c8ceb2a97dec7c1cd5ebbfca8e6b90553394021e73854a018ceb59c9cc" Sep 08 17:21:49.690342 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:49.690303 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c103c8ceb2a97dec7c1cd5ebbfca8e6b90553394021e73854a018ceb59c9cc"} err="failed to get container status \"71c103c8ceb2a97dec7c1cd5ebbfca8e6b90553394021e73854a018ceb59c9cc\": rpc error: code = NotFound desc = could not find container \"71c103c8ceb2a97dec7c1cd5ebbfca8e6b90553394021e73854a018ceb59c9cc\": container with ID starting with 71c103c8ceb2a97dec7c1cd5ebbfca8e6b90553394021e73854a018ceb59c9cc not found: ID does not exist" Sep 08 17:21:49.706155 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:49.706123 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75d5cb9587-qb5pj"] Sep 08 17:21:49.708559 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:49.708536 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75d5cb9587-qb5pj"] Sep 08 17:21:49.761744 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:49.761707 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f458df5-6b01-4d03-a598-e77744281cc9" path="/var/lib/kubelet/pods/4f458df5-6b01-4d03-a598-e77744281cc9/volumes" Sep 08 17:21:50.396239 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:50.396199 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:21:50.517377 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:50.517340 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75d5cb9587-9g8g5"] Sep 08 17:21:50.517765 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:50.517752 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f458df5-6b01-4d03-a598-e77744281cc9" containerName="manager" Sep 08 17:21:50.517811 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:50.517767 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f458df5-6b01-4d03-a598-e77744281cc9" containerName="manager" Sep 08 17:21:50.517844 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:50.517838 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f458df5-6b01-4d03-a598-e77744281cc9" containerName="manager" Sep 08 17:21:50.522255 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:50.522236 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75d5cb9587-9g8g5" Sep 08 17:21:50.526446 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:50.526382 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 08 17:21:50.526446 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:50.526390 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j7x8d\"" Sep 08 17:21:50.526446 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:50.526413 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 08 17:21:50.526673 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:50.526437 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 08 17:21:50.532872 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:50.532848 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75d5cb9587-9g8g5"] Sep 08 17:21:50.635975 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:50.635935 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14f51478-1e12-478a-91fb-f0e7d69d7447-webhook-cert\") pod \"maas-controller-75d5cb9587-9g8g5\" (UID: \"14f51478-1e12-478a-91fb-f0e7d69d7447\") " pod="opendatahub/maas-controller-75d5cb9587-9g8g5" Sep 08 17:21:50.636171 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:50.636006 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14f51478-1e12-478a-91fb-f0e7d69d7447-metrics-certs\") pod \"maas-controller-75d5cb9587-9g8g5\" (UID: \"14f51478-1e12-478a-91fb-f0e7d69d7447\") " pod="opendatahub/maas-controller-75d5cb9587-9g8g5" Sep 08 17:21:50.636171 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:50.636055 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4dgn\" (UniqueName: \"kubernetes.io/projected/14f51478-1e12-478a-91fb-f0e7d69d7447-kube-api-access-m4dgn\") pod \"maas-controller-75d5cb9587-9g8g5\" (UID: \"14f51478-1e12-478a-91fb-f0e7d69d7447\") " pod="opendatahub/maas-controller-75d5cb9587-9g8g5" Sep 08 17:21:50.736740 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:50.736707 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14f51478-1e12-478a-91fb-f0e7d69d7447-webhook-cert\") pod \"maas-controller-75d5cb9587-9g8g5\" (UID: \"14f51478-1e12-478a-91fb-f0e7d69d7447\") " pod="opendatahub/maas-controller-75d5cb9587-9g8g5" Sep 08 17:21:50.736929 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:50.736774 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14f51478-1e12-478a-91fb-f0e7d69d7447-metrics-certs\") pod \"maas-controller-75d5cb9587-9g8g5\" (UID: \"14f51478-1e12-478a-91fb-f0e7d69d7447\") " pod="opendatahub/maas-controller-75d5cb9587-9g8g5" Sep 08 17:21:50.736929 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:50.736808 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4dgn\" (UniqueName: \"kubernetes.io/projected/14f51478-1e12-478a-91fb-f0e7d69d7447-kube-api-access-m4dgn\") pod \"maas-controller-75d5cb9587-9g8g5\" (UID: \"14f51478-1e12-478a-91fb-f0e7d69d7447\") " pod="opendatahub/maas-controller-75d5cb9587-9g8g5" Sep 08 17:21:50.739290 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:50.739266 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14f51478-1e12-478a-91fb-f0e7d69d7447-webhook-cert\") pod \"maas-controller-75d5cb9587-9g8g5\" (UID: \"14f51478-1e12-478a-91fb-f0e7d69d7447\") " pod="opendatahub/maas-controller-75d5cb9587-9g8g5" Sep 08 17:21:50.739411 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:50.739270 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14f51478-1e12-478a-91fb-f0e7d69d7447-metrics-certs\") pod \"maas-controller-75d5cb9587-9g8g5\" (UID: \"14f51478-1e12-478a-91fb-f0e7d69d7447\") " pod="opendatahub/maas-controller-75d5cb9587-9g8g5" Sep 08 17:21:50.746139 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:50.746117 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4dgn\" (UniqueName: \"kubernetes.io/projected/14f51478-1e12-478a-91fb-f0e7d69d7447-kube-api-access-m4dgn\") pod \"maas-controller-75d5cb9587-9g8g5\" (UID: \"14f51478-1e12-478a-91fb-f0e7d69d7447\") " pod="opendatahub/maas-controller-75d5cb9587-9g8g5" Sep 08 17:21:50.834262 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:50.834223 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75d5cb9587-9g8g5" Sep 08 17:21:51.119604 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:51.119574 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75d5cb9587-9g8g5"] Sep 08 17:21:51.121629 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:21:51.121601 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f51478_1e12_478a_91fb_f0e7d69d7447.slice/crio-12870a802dcf1b3c6622f92175b09aab422075d7bbfd3dac617ea2ea716a6dcf WatchSource:0}: Error finding container 12870a802dcf1b3c6622f92175b09aab422075d7bbfd3dac617ea2ea716a6dcf: Status 404 returned error can't find the container with id 12870a802dcf1b3c6622f92175b09aab422075d7bbfd3dac617ea2ea716a6dcf Sep 08 17:21:51.700225 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:51.700180 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75d5cb9587-9g8g5" event={"ID":"14f51478-1e12-478a-91fb-f0e7d69d7447","Type":"ContainerStarted","Data":"d9922a9c76abbac3067e2c31e744d1841476f5faab8377a2357ce06187dd8196"} Sep 08 17:21:51.700703 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:51.700240 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75d5cb9587-9g8g5" event={"ID":"14f51478-1e12-478a-91fb-f0e7d69d7447","Type":"ContainerStarted","Data":"12870a802dcf1b3c6622f92175b09aab422075d7bbfd3dac617ea2ea716a6dcf"} Sep 08 17:21:51.700703 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:51.700378 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75d5cb9587-9g8g5" Sep 08 17:21:51.723436 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:21:51.723371 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-75d5cb9587-9g8g5" podStartSLOduration=1.282437846 podStartE2EDuration="1.723352739s" totalImagesPullingTime="440.914893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:21:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:21:51.720783626 +0000 UTC m=+1874.457614743" watchObservedRunningTime="2026-09-08 17:21:51.723352739 +0000 UTC m=+1874.460183855" Sep 08 17:22:02.708675 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:22:02.708597 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75d5cb9587-9g8g5" Sep 08 17:22:22.337159 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:22:22.337123 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:23:12.495125 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:23:12.495068 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t"] Sep 08 17:23:12.495904 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:23:12.495348 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t" podUID="82b588c3-bcc8-4877-90cf-579a4e5eb5b3" containerName="manager" containerID="cri-o://44765351b0e6362bb1e7e1d9c3cfec99ae05200ee2ce3ad5555f55f1a28d4043" gracePeriod=10 Sep 08 17:23:12.745988 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:23:12.745917 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t" Sep 08 17:23:12.845488 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:23:12.845450 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82b588c3-bcc8-4877-90cf-579a4e5eb5b3-extensions-socket-volume\") pod \"82b588c3-bcc8-4877-90cf-579a4e5eb5b3\" (UID: \"82b588c3-bcc8-4877-90cf-579a4e5eb5b3\") " Sep 08 17:23:12.845640 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:23:12.845508 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wdlf\" (UniqueName: \"kubernetes.io/projected/82b588c3-bcc8-4877-90cf-579a4e5eb5b3-kube-api-access-9wdlf\") pod \"82b588c3-bcc8-4877-90cf-579a4e5eb5b3\" (UID: \"82b588c3-bcc8-4877-90cf-579a4e5eb5b3\") " Sep 08 17:23:12.845932 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:23:12.845906 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b588c3-bcc8-4877-90cf-579a4e5eb5b3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "82b588c3-bcc8-4877-90cf-579a4e5eb5b3" (UID: "82b588c3-bcc8-4877-90cf-579a4e5eb5b3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:23:12.847597 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:23:12.847576 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b588c3-bcc8-4877-90cf-579a4e5eb5b3-kube-api-access-9wdlf" (OuterVolumeSpecName: "kube-api-access-9wdlf") pod "82b588c3-bcc8-4877-90cf-579a4e5eb5b3" (UID: "82b588c3-bcc8-4877-90cf-579a4e5eb5b3"). InnerVolumeSpecName "kube-api-access-9wdlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:23:12.947057 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:23:12.947017 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82b588c3-bcc8-4877-90cf-579a4e5eb5b3-extensions-socket-volume\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:23:12.947057 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:23:12.947049 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wdlf\" (UniqueName: \"kubernetes.io/projected/82b588c3-bcc8-4877-90cf-579a4e5eb5b3-kube-api-access-9wdlf\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:23:13.172002 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:23:13.171966 2565 generic.go:356] "Generic (PLEG): container finished" podID="82b588c3-bcc8-4877-90cf-579a4e5eb5b3" containerID="44765351b0e6362bb1e7e1d9c3cfec99ae05200ee2ce3ad5555f55f1a28d4043" exitCode=0 Sep 08 17:23:13.172167 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:23:13.172012 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t" Sep 08 17:23:13.172167 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:23:13.172046 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t" event={"ID":"82b588c3-bcc8-4877-90cf-579a4e5eb5b3","Type":"ContainerDied","Data":"44765351b0e6362bb1e7e1d9c3cfec99ae05200ee2ce3ad5555f55f1a28d4043"} Sep 08 17:23:13.172167 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:23:13.172100 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t" event={"ID":"82b588c3-bcc8-4877-90cf-579a4e5eb5b3","Type":"ContainerDied","Data":"4f15a420dee0482b2ff6a482f248225229f1833609731adc04bbdd6f05b71077"} Sep 08 17:23:13.172167 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:23:13.172122 2565 scope.go:117] "RemoveContainer" containerID="44765351b0e6362bb1e7e1d9c3cfec99ae05200ee2ce3ad5555f55f1a28d4043" Sep 08 17:23:13.182378 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:23:13.182356 2565 scope.go:117] "RemoveContainer" containerID="44765351b0e6362bb1e7e1d9c3cfec99ae05200ee2ce3ad5555f55f1a28d4043" Sep 08 17:23:13.182703 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:23:13.182681 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44765351b0e6362bb1e7e1d9c3cfec99ae05200ee2ce3ad5555f55f1a28d4043\": container with ID starting with 44765351b0e6362bb1e7e1d9c3cfec99ae05200ee2ce3ad5555f55f1a28d4043 not found: ID does not exist" containerID="44765351b0e6362bb1e7e1d9c3cfec99ae05200ee2ce3ad5555f55f1a28d4043" Sep 08 17:23:13.182771 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:23:13.182713 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44765351b0e6362bb1e7e1d9c3cfec99ae05200ee2ce3ad5555f55f1a28d4043"} err="failed to get container status \"44765351b0e6362bb1e7e1d9c3cfec99ae05200ee2ce3ad5555f55f1a28d4043\": rpc error: code = NotFound desc = could not find container \"44765351b0e6362bb1e7e1d9c3cfec99ae05200ee2ce3ad5555f55f1a28d4043\": container with ID starting with 44765351b0e6362bb1e7e1d9c3cfec99ae05200ee2ce3ad5555f55f1a28d4043 not found: ID does not exist" Sep 08 17:23:13.198111 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:23:13.198075 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t"] Sep 08 17:23:13.205461 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:23:13.205414 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cm29t"] Sep 08 17:23:13.759716 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:23:13.759679 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82b588c3-bcc8-4877-90cf-579a4e5eb5b3" path="/var/lib/kubelet/pods/82b588c3-bcc8-4877-90cf-579a4e5eb5b3/volumes" Sep 08 17:24:13.157959 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:24:13.157919 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gb4wg"] Sep 08 17:24:13.158535 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:24:13.158316 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82b588c3-bcc8-4877-90cf-579a4e5eb5b3" containerName="manager" Sep 08 17:24:13.158535 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:24:13.158328 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="82b588c3-bcc8-4877-90cf-579a4e5eb5b3" containerName="manager" Sep 08 17:24:13.158535 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:24:13.158389 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="82b588c3-bcc8-4877-90cf-579a4e5eb5b3" containerName="manager" Sep 08 17:24:13.161412 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:24:13.161396 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gb4wg" Sep 08 17:24:13.164185 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:24:13.164164 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jgzqv\"" Sep 08 17:24:13.176366 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:24:13.176341 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gb4wg"] Sep 08 17:24:13.276417 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:24:13.276379 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4056d2c5-70e0-4368-a5e7-21fb1b3bffb5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gb4wg\" (UID: \"4056d2c5-70e0-4368-a5e7-21fb1b3bffb5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gb4wg" Sep 08 17:24:13.276633 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:24:13.276473 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htjf7\" (UniqueName: \"kubernetes.io/projected/4056d2c5-70e0-4368-a5e7-21fb1b3bffb5-kube-api-access-htjf7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gb4wg\" (UID: \"4056d2c5-70e0-4368-a5e7-21fb1b3bffb5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gb4wg" Sep 08 17:24:13.377219 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:24:13.377176 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4056d2c5-70e0-4368-a5e7-21fb1b3bffb5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gb4wg\" (UID: \"4056d2c5-70e0-4368-a5e7-21fb1b3bffb5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gb4wg" Sep 08 17:24:13.377397 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:24:13.377240 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htjf7\" (UniqueName: \"kubernetes.io/projected/4056d2c5-70e0-4368-a5e7-21fb1b3bffb5-kube-api-access-htjf7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gb4wg\" (UID: \"4056d2c5-70e0-4368-a5e7-21fb1b3bffb5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gb4wg" Sep 08 17:24:13.377604 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:24:13.377583 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4056d2c5-70e0-4368-a5e7-21fb1b3bffb5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gb4wg\" (UID: \"4056d2c5-70e0-4368-a5e7-21fb1b3bffb5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gb4wg" Sep 08 17:24:13.391572 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:24:13.391542 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htjf7\" (UniqueName: \"kubernetes.io/projected/4056d2c5-70e0-4368-a5e7-21fb1b3bffb5-kube-api-access-htjf7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gb4wg\" (UID: \"4056d2c5-70e0-4368-a5e7-21fb1b3bffb5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gb4wg" Sep 08 17:24:13.471608 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:24:13.471562 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gb4wg" Sep 08 17:24:13.774079 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:24:13.773980 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gb4wg"] Sep 08 17:24:13.779945 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:24:13.779916 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4056d2c5_70e0_4368_a5e7_21fb1b3bffb5.slice/crio-e500d6dd715737f3ffe12cbbbad7f28e9adafec426b19884add55fa35c8ecbc9 WatchSource:0}: Error finding container e500d6dd715737f3ffe12cbbbad7f28e9adafec426b19884add55fa35c8ecbc9: Status 404 returned error can't find the container with id e500d6dd715737f3ffe12cbbbad7f28e9adafec426b19884add55fa35c8ecbc9 Sep 08 17:24:14.519762 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:24:14.519717 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gb4wg" event={"ID":"4056d2c5-70e0-4368-a5e7-21fb1b3bffb5","Type":"ContainerStarted","Data":"0a98c490bb90611d54ea8502ed9212ad73a844a930da462cd45c9f48aa438ad3"} Sep 08 17:24:14.520156 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:24:14.519788 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gb4wg" event={"ID":"4056d2c5-70e0-4368-a5e7-21fb1b3bffb5","Type":"ContainerStarted","Data":"e500d6dd715737f3ffe12cbbbad7f28e9adafec426b19884add55fa35c8ecbc9"} Sep 08 17:24:14.520156 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:24:14.519842 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gb4wg" Sep 08 17:24:14.543713 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:24:14.543660 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gb4wg" podStartSLOduration=1.543643603 podStartE2EDuration="1.543643603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:24:14.539686723 +0000 UTC m=+2017.276517840" watchObservedRunningTime="2026-09-08 17:24:14.543643603 +0000 UTC m=+2017.280474720" Sep 08 17:24:25.528202 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:24:25.528168 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gb4wg" Sep 08 17:24:57.251045 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:24:57.251004 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:25:09.516103 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:25:09.516013 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:25:36.151728 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:25:36.151693 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:25:36.251413 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:25:36.251374 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:25:37.129157 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:25:37.129116 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:25:37.948835 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:25:37.948798 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/2.log" Sep 08 17:25:37.958099 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:25:37.958074 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xxnl_bf53ac63-f864-402b-bf12-45b289e7e13d/ovn-acl-logging/0.log" Sep 08 17:25:37.964326 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:25:37.964305 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/2.log" Sep 08 17:25:37.972705 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:25:37.972683 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9xxnl_bf53ac63-f864-402b-bf12-45b289e7e13d/ovn-acl-logging/0.log" Sep 08 17:25:39.308744 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:25:39.308710 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:26:02.516619 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:02.516580 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79cd597bc5-m82fx"] Sep 08 17:26:02.520339 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:02.520312 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-m82fx" Sep 08 17:26:02.533292 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:02.533263 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79cd597bc5-m82fx"] Sep 08 17:26:02.644301 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:02.644264 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjrwh\" (UniqueName: \"kubernetes.io/projected/c172fcf3-de02-46aa-ab57-c259889c1ff7-kube-api-access-mjrwh\") pod \"maas-api-79cd597bc5-m82fx\" (UID: \"c172fcf3-de02-46aa-ab57-c259889c1ff7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-m82fx" Sep 08 17:26:02.644498 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:02.644312 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c172fcf3-de02-46aa-ab57-c259889c1ff7-metrics-certs\") pod \"maas-api-79cd597bc5-m82fx\" (UID: \"c172fcf3-de02-46aa-ab57-c259889c1ff7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-m82fx" Sep 08 17:26:02.644498 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:02.644387 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c172fcf3-de02-46aa-ab57-c259889c1ff7-openshift-service-ca\") pod \"maas-api-79cd597bc5-m82fx\" (UID: \"c172fcf3-de02-46aa-ab57-c259889c1ff7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-m82fx" Sep 08 17:26:02.644498 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:02.644478 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c172fcf3-de02-46aa-ab57-c259889c1ff7-maas-api-tls\") pod \"maas-api-79cd597bc5-m82fx\" (UID: \"c172fcf3-de02-46aa-ab57-c259889c1ff7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-m82fx" Sep 08 17:26:02.745285 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:02.745248 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c172fcf3-de02-46aa-ab57-c259889c1ff7-openshift-service-ca\") pod \"maas-api-79cd597bc5-m82fx\" (UID: \"c172fcf3-de02-46aa-ab57-c259889c1ff7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-m82fx" Sep 08 17:26:02.745285 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:02.745292 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c172fcf3-de02-46aa-ab57-c259889c1ff7-maas-api-tls\") pod \"maas-api-79cd597bc5-m82fx\" (UID: \"c172fcf3-de02-46aa-ab57-c259889c1ff7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-m82fx" Sep 08 17:26:02.745570 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:02.745360 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjrwh\" (UniqueName: \"kubernetes.io/projected/c172fcf3-de02-46aa-ab57-c259889c1ff7-kube-api-access-mjrwh\") pod \"maas-api-79cd597bc5-m82fx\" (UID: \"c172fcf3-de02-46aa-ab57-c259889c1ff7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-m82fx" Sep 08 17:26:02.745570 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:02.745385 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c172fcf3-de02-46aa-ab57-c259889c1ff7-metrics-certs\") pod \"maas-api-79cd597bc5-m82fx\" (UID: \"c172fcf3-de02-46aa-ab57-c259889c1ff7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-m82fx" Sep 08 17:26:02.746163 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:02.746132 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c172fcf3-de02-46aa-ab57-c259889c1ff7-openshift-service-ca\") pod \"maas-api-79cd597bc5-m82fx\" (UID: \"c172fcf3-de02-46aa-ab57-c259889c1ff7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-m82fx" Sep 08 17:26:02.747929 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:02.747896 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c172fcf3-de02-46aa-ab57-c259889c1ff7-metrics-certs\") pod \"maas-api-79cd597bc5-m82fx\" (UID: \"c172fcf3-de02-46aa-ab57-c259889c1ff7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-m82fx" Sep 08 17:26:02.748168 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:02.748146 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c172fcf3-de02-46aa-ab57-c259889c1ff7-maas-api-tls\") pod \"maas-api-79cd597bc5-m82fx\" (UID: \"c172fcf3-de02-46aa-ab57-c259889c1ff7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-m82fx" Sep 08 17:26:02.755938 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:02.755912 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjrwh\" (UniqueName: \"kubernetes.io/projected/c172fcf3-de02-46aa-ab57-c259889c1ff7-kube-api-access-mjrwh\") pod \"maas-api-79cd597bc5-m82fx\" (UID: \"c172fcf3-de02-46aa-ab57-c259889c1ff7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-m82fx" Sep 08 17:26:02.833794 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:02.833706 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-m82fx" Sep 08 17:26:03.112711 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:03.112685 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79cd597bc5-m82fx"] Sep 08 17:26:03.114720 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:26:03.114690 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc172fcf3_de02_46aa_ab57_c259889c1ff7.slice/crio-77543222adf31ddc4ee973fcb51f6fee3860849455b54635c43eff80d1b6e9ac WatchSource:0}: Error finding container 77543222adf31ddc4ee973fcb51f6fee3860849455b54635c43eff80d1b6e9ac: Status 404 returned error can't find the container with id 77543222adf31ddc4ee973fcb51f6fee3860849455b54635c43eff80d1b6e9ac Sep 08 17:26:03.150242 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:03.150201 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-m82fx" event={"ID":"c172fcf3-de02-46aa-ab57-c259889c1ff7","Type":"ContainerStarted","Data":"77543222adf31ddc4ee973fcb51f6fee3860849455b54635c43eff80d1b6e9ac"} Sep 08 17:26:04.157305 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:04.157269 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-m82fx" event={"ID":"c172fcf3-de02-46aa-ab57-c259889c1ff7","Type":"ContainerStarted","Data":"2299644404f842912d51165af3e89803037b12daf3742eafa92f772f0eeab78d"} Sep 08 17:26:04.157762 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:04.157460 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-m82fx" Sep 08 17:26:04.173909 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:04.173854 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-m82fx" podStartSLOduration=1.736711533 podStartE2EDuration="2.173838236s" totalImagesPullingTime="437.126703ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:26:04.172402804 +0000 UTC m=+2126.909233923" watchObservedRunningTime="2026-09-08 17:26:04.173838236 +0000 UTC m=+2126.910669351" Sep 08 17:26:10.169144 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:10.169112 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-m82fx" Sep 08 17:26:17.761371 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:17.761336 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b"] Sep 08 17:26:17.763945 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:17.761528 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" podUID="ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2" containerName="maas-api" containerID="cri-o://dc19028dfb187f19f88167356eb2a556a01c07c07cbc2315785b6470ea3c552b" gracePeriod=30 Sep 08 17:26:19.017946 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.017917 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" Sep 08 17:26:19.097865 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.097755 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-openshift-service-ca\") pod \"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2\" (UID: \"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2\") " Sep 08 17:26:19.097865 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.097816 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-maas-api-tls\") pod \"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2\" (UID: \"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2\") " Sep 08 17:26:19.097865 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.097848 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-metrics-certs\") pod \"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2\" (UID: \"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2\") " Sep 08 17:26:19.097865 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.097866 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcsnk\" (UniqueName: \"kubernetes.io/projected/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-kube-api-access-jcsnk\") pod \"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2\" (UID: \"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2\") " Sep 08 17:26:19.098251 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.098223 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2" (UID: "ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:26:19.100158 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.100134 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2" (UID: "ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:26:19.100244 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.100164 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2" (UID: "ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:26:19.100244 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.100208 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-kube-api-access-jcsnk" (OuterVolumeSpecName: "kube-api-access-jcsnk") pod "ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2" (UID: "ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2"). InnerVolumeSpecName "kube-api-access-jcsnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:26:19.198615 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.198577 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-openshift-service-ca\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:26:19.198615 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.198612 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-maas-api-tls\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:26:19.198615 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.198621 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-metrics-certs\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:26:19.198862 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.198629 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcsnk\" (UniqueName: \"kubernetes.io/projected/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2-kube-api-access-jcsnk\") on node \"ip-10-0-137-108.ec2.internal\" DevicePath \"\"" Sep 08 17:26:19.254258 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.254212 2565 generic.go:356] "Generic (PLEG): container finished" podID="ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2" containerID="dc19028dfb187f19f88167356eb2a556a01c07c07cbc2315785b6470ea3c552b" exitCode=0 Sep 08 17:26:19.254258 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.254263 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" event={"ID":"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2","Type":"ContainerDied","Data":"dc19028dfb187f19f88167356eb2a556a01c07c07cbc2315785b6470ea3c552b"} Sep 08 17:26:19.254574 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.254307 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" event={"ID":"ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2","Type":"ContainerDied","Data":"cb3ff11a2d606a652441c8f22dd5475ff75b8d1102bf74a699f4853f886dad4c"} Sep 08 17:26:19.254574 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.254329 2565 scope.go:117] "RemoveContainer" containerID="dc19028dfb187f19f88167356eb2a556a01c07c07cbc2315785b6470ea3c552b" Sep 08 17:26:19.254574 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.254265 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b" Sep 08 17:26:19.264468 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.264444 2565 scope.go:117] "RemoveContainer" containerID="dc19028dfb187f19f88167356eb2a556a01c07c07cbc2315785b6470ea3c552b" Sep 08 17:26:19.264813 ip-10-0-137-108 kubenswrapper[2565]: E0908 17:26:19.264791 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc19028dfb187f19f88167356eb2a556a01c07c07cbc2315785b6470ea3c552b\": container with ID starting with dc19028dfb187f19f88167356eb2a556a01c07c07cbc2315785b6470ea3c552b not found: ID does not exist" containerID="dc19028dfb187f19f88167356eb2a556a01c07c07cbc2315785b6470ea3c552b" Sep 08 17:26:19.264902 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.264822 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc19028dfb187f19f88167356eb2a556a01c07c07cbc2315785b6470ea3c552b"} err="failed to get container status \"dc19028dfb187f19f88167356eb2a556a01c07c07cbc2315785b6470ea3c552b\": rpc error: code = NotFound desc = could not find container \"dc19028dfb187f19f88167356eb2a556a01c07c07cbc2315785b6470ea3c552b\": container with ID starting with dc19028dfb187f19f88167356eb2a556a01c07c07cbc2315785b6470ea3c552b not found: ID does not exist" Sep 08 17:26:19.275552 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.275518 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b"] Sep 08 17:26:19.279592 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.279561 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-lt57b"] Sep 08 17:26:19.761076 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:19.761037 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2" path="/var/lib/kubelet/pods/ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2/volumes" Sep 08 17:26:32.246926 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:32.246816 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-755dd4cb9c-7nrw6_5693acfa-95c2-4930-bf0f-c56c38438c7d/authorino/0.log" Sep 08 17:26:37.595747 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:37.595714 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-s8z86_517d507a-eadc-4cf7-9193-9b9bdfcefe66/manager/0.log" Sep 08 17:26:37.713038 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:37.713006 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-q788k_d2bd1bab-bda5-45cf-8812-2a3e6b2026ba/manager/0.log" Sep 08 17:26:37.830771 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:37.830740 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75d5cb9587-9g8g5_14f51478-1e12-478a-91fb-f0e7d69d7447/manager/0.log" Sep 08 17:26:38.175689 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:38.175661 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84f58c54b-62m72_4490dbae-ceb9-4f64-a491-fc2f20ecc243/copy-manifests/0.log" Sep 08 17:26:38.181844 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:38.181818 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84f58c54b-62m72_4490dbae-ceb9-4f64-a491-fc2f20ecc243/manager/0.log" Sep 08 17:26:38.872147 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:38.872109 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79cd597bc5-m82fx_c172fcf3-de02-46aa-ab57-c259889c1ff7/maas-api/0.log" Sep 08 17:26:39.200206 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:39.200177 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-657vn_fccab80a-e7c5-4811-ba95-294ac01ab453/postgres/0.log" Sep 08 17:26:39.967596 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:39.967556 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-755dd4cb9c-7nrw6_5693acfa-95c2-4930-bf0f-c56c38438c7d/authorino/0.log" Sep 08 17:26:40.077659 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:40.077630 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-x9b85_b3c4cd20-2303-48e9-9c7b-dd655487489e/manager/0.log" Sep 08 17:26:40.187387 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:40.187352 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-c6fk2_1e6fef87-3f71-4022-9488-de1a691d34e3/manager/0.log" Sep 08 17:26:40.301559 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:40.301486 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-dm7hb_da60a5c3-4a54-4079-9218-842083016fa5/kuadrant-console-plugin/0.log" Sep 08 17:26:40.422611 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:40.422585 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-gb4wg_4056d2c5-70e0-4368-a5e7-21fb1b3bffb5/manager/0.log" Sep 08 17:26:40.648027 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:40.647948 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-fn9mx_8b11eb00-7c73-4334-82cc-bc64fc094fa7/manager/0.log" Sep 08 17:26:41.223287 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:41.223259 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d8677cc5f-pcp9g_7e359e34-b048-4110-9fc0-149b12c3363a/kube-auth-proxy/0.log" Sep 08 17:26:41.707585 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:41.707548 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-vrwnc_f5cd24d0-2d8f-4639-9fac-d4ae2ba1676d/payload-processing/0.log" Sep 08 17:26:41.844148 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:41.844125 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-858549dccb-6vpw7_ec6cae4f-377a-4717-acb8-63166637113a/router/0.log" Sep 08 17:26:42.290076 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:42.290048 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-lt675_e19ac764-d081-445b-a44c-c77ca448cdbe/main/0.log" Sep 08 17:26:42.717533 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:42.717498 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-22pzf_0b011a03-62bd-473e-bbb6-4863b92ad9d8/main/0.log" Sep 08 17:26:47.886525 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:47.886488 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7djdj/must-gather-6brz6"] Sep 08 17:26:47.887064 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:47.887041 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2" containerName="maas-api" Sep 08 17:26:47.887064 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:47.887066 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2" containerName="maas-api" Sep 08 17:26:47.887250 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:47.887231 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea64a9f3-4c5d-4ba4-9535-cd0f4774e9e2" containerName="maas-api" Sep 08 17:26:47.890482 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:47.890458 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7djdj/must-gather-6brz6" Sep 08 17:26:47.892797 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:47.892774 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7djdj\"/\"openshift-service-ca.crt\"" Sep 08 17:26:47.892896 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:47.892773 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7djdj\"/\"kube-root-ca.crt\"" Sep 08 17:26:47.893404 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:47.893390 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7djdj\"/\"default-dockercfg-q2jfr\"" Sep 08 17:26:47.899132 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:47.899107 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7djdj/must-gather-6brz6"] Sep 08 17:26:47.900924 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:47.900902 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:26:47.941966 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:47.941939 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkq67\" (UniqueName: \"kubernetes.io/projected/c653a484-f90e-4e1a-a6fc-4f423e6bc189-kube-api-access-rkq67\") pod \"must-gather-6brz6\" (UID: \"c653a484-f90e-4e1a-a6fc-4f423e6bc189\") " pod="openshift-must-gather-7djdj/must-gather-6brz6" Sep 08 17:26:47.942072 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:47.941986 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c653a484-f90e-4e1a-a6fc-4f423e6bc189-must-gather-output\") pod \"must-gather-6brz6\" (UID: \"c653a484-f90e-4e1a-a6fc-4f423e6bc189\") " pod="openshift-must-gather-7djdj/must-gather-6brz6" Sep 08 17:26:48.000893 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:48.000863 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" limit=1024 Sep 08 17:26:48.042998 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:48.042976 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkq67\" (UniqueName: \"kubernetes.io/projected/c653a484-f90e-4e1a-a6fc-4f423e6bc189-kube-api-access-rkq67\") pod \"must-gather-6brz6\" (UID: \"c653a484-f90e-4e1a-a6fc-4f423e6bc189\") " pod="openshift-must-gather-7djdj/must-gather-6brz6" Sep 08 17:26:48.043091 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:48.043016 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c653a484-f90e-4e1a-a6fc-4f423e6bc189-must-gather-output\") pod \"must-gather-6brz6\" (UID: \"c653a484-f90e-4e1a-a6fc-4f423e6bc189\") " pod="openshift-must-gather-7djdj/must-gather-6brz6" Sep 08 17:26:48.043314 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:48.043299 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c653a484-f90e-4e1a-a6fc-4f423e6bc189-must-gather-output\") pod \"must-gather-6brz6\" (UID: \"c653a484-f90e-4e1a-a6fc-4f423e6bc189\") " pod="openshift-must-gather-7djdj/must-gather-6brz6" Sep 08 17:26:48.051015 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:48.050992 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkq67\" (UniqueName: \"kubernetes.io/projected/c653a484-f90e-4e1a-a6fc-4f423e6bc189-kube-api-access-rkq67\") pod \"must-gather-6brz6\" (UID: \"c653a484-f90e-4e1a-a6fc-4f423e6bc189\") " pod="openshift-must-gather-7djdj/must-gather-6brz6" Sep 08 17:26:48.200153 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:48.200128 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7djdj/must-gather-6brz6" Sep 08 17:26:48.470553 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:48.470528 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7djdj/must-gather-6brz6"] Sep 08 17:26:48.472579 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:26:48.472551 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc653a484_f90e_4e1a_a6fc_4f423e6bc189.slice/crio-2cf08340119586ee15a9235be5827035b232f4c9a18af13300de9c7d588aac6e WatchSource:0}: Error finding container 2cf08340119586ee15a9235be5827035b232f4c9a18af13300de9c7d588aac6e: Status 404 returned error can't find the container with id 2cf08340119586ee15a9235be5827035b232f4c9a18af13300de9c7d588aac6e Sep 08 17:26:48.474493 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:48.474477 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 17:26:49.442618 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:49.442553 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7djdj/must-gather-6brz6" event={"ID":"c653a484-f90e-4e1a-a6fc-4f423e6bc189","Type":"ContainerStarted","Data":"2cf08340119586ee15a9235be5827035b232f4c9a18af13300de9c7d588aac6e"} Sep 08 17:26:50.454465 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:50.454398 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7djdj/must-gather-6brz6" event={"ID":"c653a484-f90e-4e1a-a6fc-4f423e6bc189","Type":"ContainerStarted","Data":"e3f0ced2a63af9153f76dfbd1b23bedcf073df325ef9cddf66619965c15b44af"} Sep 08 17:26:50.455021 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:50.454488 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7djdj/must-gather-6brz6" event={"ID":"c653a484-f90e-4e1a-a6fc-4f423e6bc189","Type":"ContainerStarted","Data":"f0eadd4bfb3d15a83007a9f4cbb0261d89f91c765c4d22fc972c5a0ae271c65c"} Sep 08 17:26:50.472261 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:50.472202 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7djdj/must-gather-6brz6" podStartSLOduration=2.546055829 podStartE2EDuration="3.472181485s" totalImagesPullingTime="926.125656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:26:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:26:50.468710818 +0000 UTC m=+2173.205541935" watchObservedRunningTime="2026-09-08 17:26:50.472181485 +0000 UTC m=+2173.209012603" Sep 08 17:26:50.955980 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:50.955948 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jpx5h_192b1a00-b435-4793-98d5-d51cf7b036d1/global-pull-secret-syncer/0.log" Sep 08 17:26:51.050640 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:51.050612 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2ccgf_a6d1c7fb-9d63-4859-8ac1-46819d815405/konnectivity-agent/0.log" Sep 08 17:26:51.190768 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:51.190736 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-108.ec2.internal_209ac989f5bdb50086e7e268153789b2/haproxy/0.log" Sep 08 17:26:55.875446 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:55.875078 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-755dd4cb9c-7nrw6_5693acfa-95c2-4930-bf0f-c56c38438c7d/authorino/0.log" Sep 08 17:26:55.914381 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:55.914343 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-x9b85_b3c4cd20-2303-48e9-9c7b-dd655487489e/manager/0.log" Sep 08 17:26:55.939023 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:55.938989 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-c6fk2_1e6fef87-3f71-4022-9488-de1a691d34e3/manager/0.log" Sep 08 17:26:55.962323 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:55.962291 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-dm7hb_da60a5c3-4a54-4079-9218-842083016fa5/kuadrant-console-plugin/0.log" Sep 08 17:26:56.003736 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:56.003698 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-gb4wg_4056d2c5-70e0-4368-a5e7-21fb1b3bffb5/manager/0.log" Sep 08 17:26:56.104069 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:56.104035 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-fn9mx_8b11eb00-7c73-4334-82cc-bc64fc094fa7/manager/0.log" Sep 08 17:26:57.587861 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:57.587819 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-ngvwg_0d648a28-280d-4584-a061-704134b80708/cluster-monitoring-operator/0.log" Sep 08 17:26:57.820817 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:57.820783 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kp8dq_dde9ca85-937b-4638-89ba-60ac40e4a5d9/node-exporter/0.log" Sep 08 17:26:57.841131 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:57.841047 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kp8dq_dde9ca85-937b-4638-89ba-60ac40e4a5d9/kube-rbac-proxy/0.log" Sep 08 17:26:57.859621 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:57.859589 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kp8dq_dde9ca85-937b-4638-89ba-60ac40e4a5d9/init-textfile/0.log" Sep 08 17:26:58.288273 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:26:58.288184 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-kq45n_6d3760ef-42a3-405e-b446-ed0a9c68ffa6/prometheus-operator-admission-webhook/0.log" Sep 08 17:27:00.582114 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:00.582041 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/2.log" Sep 08 17:27:00.591626 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:00.591598 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8xjm5_5977d6b9-f03a-40f2-bdab-9f07d1065c2d/console-operator/3.log" Sep 08 17:27:01.125207 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.125177 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-b9cc75d5d-4766s_187a2cde-e262-4c91-94ab-dabffecb5b69/console/0.log" Sep 08 17:27:01.172824 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.172789 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85"] Sep 08 17:27:01.176542 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.176512 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-rkvsz_b8eda268-da49-429e-bba5-0d983fcb4a74/download-server/0.log" Sep 08 17:27:01.178932 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.178906 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85" Sep 08 17:27:01.184471 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.184440 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85"] Sep 08 17:27:01.290043 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.290004 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14f52f93-daf7-4c2a-8d80-f765daba44a4-proc\") pod \"perf-node-gather-daemonset-9pn85\" (UID: \"14f52f93-daf7-4c2a-8d80-f765daba44a4\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85" Sep 08 17:27:01.290319 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.290295 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14f52f93-daf7-4c2a-8d80-f765daba44a4-lib-modules\") pod \"perf-node-gather-daemonset-9pn85\" (UID: \"14f52f93-daf7-4c2a-8d80-f765daba44a4\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85" Sep 08 17:27:01.290549 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.290518 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14f52f93-daf7-4c2a-8d80-f765daba44a4-sys\") pod \"perf-node-gather-daemonset-9pn85\" (UID: \"14f52f93-daf7-4c2a-8d80-f765daba44a4\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85" Sep 08 17:27:01.290646 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.290563 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzqw2\" (UniqueName: \"kubernetes.io/projected/14f52f93-daf7-4c2a-8d80-f765daba44a4-kube-api-access-jzqw2\") pod \"perf-node-gather-daemonset-9pn85\" (UID: \"14f52f93-daf7-4c2a-8d80-f765daba44a4\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85" Sep 08 17:27:01.290708 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.290650 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14f52f93-daf7-4c2a-8d80-f765daba44a4-podres\") pod \"perf-node-gather-daemonset-9pn85\" (UID: \"14f52f93-daf7-4c2a-8d80-f765daba44a4\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85" Sep 08 17:27:01.392173 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.392082 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14f52f93-daf7-4c2a-8d80-f765daba44a4-proc\") pod \"perf-node-gather-daemonset-9pn85\" (UID: \"14f52f93-daf7-4c2a-8d80-f765daba44a4\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85" Sep 08 17:27:01.392173 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.392159 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14f52f93-daf7-4c2a-8d80-f765daba44a4-lib-modules\") pod \"perf-node-gather-daemonset-9pn85\" (UID: \"14f52f93-daf7-4c2a-8d80-f765daba44a4\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85" Sep 08 17:27:01.392416 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.392200 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14f52f93-daf7-4c2a-8d80-f765daba44a4-proc\") pod \"perf-node-gather-daemonset-9pn85\" (UID: \"14f52f93-daf7-4c2a-8d80-f765daba44a4\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85" Sep 08 17:27:01.392416 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.392260 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14f52f93-daf7-4c2a-8d80-f765daba44a4-sys\") pod \"perf-node-gather-daemonset-9pn85\" (UID: \"14f52f93-daf7-4c2a-8d80-f765daba44a4\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85" Sep 08 17:27:01.392416 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.392288 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzqw2\" (UniqueName: \"kubernetes.io/projected/14f52f93-daf7-4c2a-8d80-f765daba44a4-kube-api-access-jzqw2\") pod \"perf-node-gather-daemonset-9pn85\" (UID: \"14f52f93-daf7-4c2a-8d80-f765daba44a4\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85" Sep 08 17:27:01.392416 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.392332 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14f52f93-daf7-4c2a-8d80-f765daba44a4-podres\") pod \"perf-node-gather-daemonset-9pn85\" (UID: \"14f52f93-daf7-4c2a-8d80-f765daba44a4\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85" Sep 08 17:27:01.392416 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.392355 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14f52f93-daf7-4c2a-8d80-f765daba44a4-sys\") pod \"perf-node-gather-daemonset-9pn85\" (UID: \"14f52f93-daf7-4c2a-8d80-f765daba44a4\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85" Sep 08 17:27:01.392720 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.392485 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14f52f93-daf7-4c2a-8d80-f765daba44a4-podres\") pod \"perf-node-gather-daemonset-9pn85\" (UID: \"14f52f93-daf7-4c2a-8d80-f765daba44a4\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85" Sep 08 17:27:01.392720 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.392355 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14f52f93-daf7-4c2a-8d80-f765daba44a4-lib-modules\") pod \"perf-node-gather-daemonset-9pn85\" (UID: \"14f52f93-daf7-4c2a-8d80-f765daba44a4\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85" Sep 08 17:27:01.400651 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.400617 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzqw2\" (UniqueName: \"kubernetes.io/projected/14f52f93-daf7-4c2a-8d80-f765daba44a4-kube-api-access-jzqw2\") pod \"perf-node-gather-daemonset-9pn85\" (UID: \"14f52f93-daf7-4c2a-8d80-f765daba44a4\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85" Sep 08 17:27:01.500284 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.500242 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85" Sep 08 17:27:01.804353 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.804316 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85"] Sep 08 17:27:01.808131 ip-10-0-137-108 kubenswrapper[2565]: W0908 17:27:01.808095 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod14f52f93_daf7_4c2a_8d80_f765daba44a4.slice/crio-ea4086577ae0a9b779e76be562d88b974decf90c5a73d86f70a43fcfcaee97aa WatchSource:0}: Error finding container ea4086577ae0a9b779e76be562d88b974decf90c5a73d86f70a43fcfcaee97aa: Status 404 returned error can't find the container with id ea4086577ae0a9b779e76be562d88b974decf90c5a73d86f70a43fcfcaee97aa Sep 08 17:27:01.886542 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:01.886514 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-2b5lx_e827254b-0e20-4391-8339-3f6541535bc9/volume-data-source-validator/0.log" Sep 08 17:27:02.601318 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:02.601270 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85" event={"ID":"14f52f93-daf7-4c2a-8d80-f765daba44a4","Type":"ContainerStarted","Data":"0cc5831c598f8ea15077d9763ebd7a8d6472a0339fcb1cab4525bc1bf218aebb"} Sep 08 17:27:02.601533 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:02.601345 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85" event={"ID":"14f52f93-daf7-4c2a-8d80-f765daba44a4","Type":"ContainerStarted","Data":"ea4086577ae0a9b779e76be562d88b974decf90c5a73d86f70a43fcfcaee97aa"} Sep 08 17:27:02.601533 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:02.601368 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85" Sep 08 17:27:02.621972 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:02.621904 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85" podStartSLOduration=1.621883798 podStartE2EDuration="1.621883798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 17:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:27:02.618858067 +0000 UTC m=+2185.355689182" watchObservedRunningTime="2026-09-08 17:27:02.621883798 +0000 UTC m=+2185.358714916" Sep 08 17:27:03.002774 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:03.002745 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-69lb7_8ed0286d-2b34-44d3-95da-2d99e0772a69/dns/0.log" Sep 08 17:27:03.022002 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:03.021975 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-69lb7_8ed0286d-2b34-44d3-95da-2d99e0772a69/kube-rbac-proxy/0.log" Sep 08 17:27:03.129079 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:03.129034 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2fv8p_1da361fa-6e95-4a71-bac4-e4e5a2c6d5ef/dns-node-resolver/0.log" Sep 08 17:27:03.757605 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:03.757571 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rr4fp_92dbe1ae-0e8e-4772-9c97-be594f062d76/node-ca/0.log" Sep 08 17:27:05.036468 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:05.029081 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d8677cc5f-pcp9g_7e359e34-b048-4110-9fc0-149b12c3363a/kube-auth-proxy/0.log" Sep 08 17:27:05.186256 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:05.186213 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-vrwnc_f5cd24d0-2d8f-4639-9fac-d4ae2ba1676d/payload-processing/0.log" Sep 08 17:27:05.208064 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:05.208032 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-858549dccb-6vpw7_ec6cae4f-377a-4717-acb8-63166637113a/router/0.log" Sep 08 17:27:06.062678 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:06.062638 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ms7mp_b46e7a48-1c3e-4499-b640-c3bd623f857d/serve-healthcheck-canary/0.log" Sep 08 17:27:06.533657 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:06.533619 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-dld4j_c1c8b5c6-12f4-451f-93ac-7e1878b355be/insights-operator/0.log" Sep 08 17:27:06.536941 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:06.536912 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-dld4j_c1c8b5c6-12f4-451f-93ac-7e1878b355be/insights-operator/1.log" Sep 08 17:27:06.687546 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:06.687515 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v57sb_18c23d36-8f8e-415c-b6fe-d08ce9316e6e/kube-rbac-proxy/0.log" Sep 08 17:27:06.706855 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:06.706823 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v57sb_18c23d36-8f8e-415c-b6fe-d08ce9316e6e/exporter/0.log" Sep 08 17:27:06.726295 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:06.726266 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v57sb_18c23d36-8f8e-415c-b6fe-d08ce9316e6e/extractor/0.log" Sep 08 17:27:08.621985 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:08.621956 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-9pn85" Sep 08 17:27:08.778089 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:08.778058 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-s8z86_517d507a-eadc-4cf7-9193-9b9bdfcefe66/manager/0.log" Sep 08 17:27:09.132556 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:09.132498 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-q788k_d2bd1bab-bda5-45cf-8812-2a3e6b2026ba/manager/0.log" Sep 08 17:27:09.205931 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:09.205894 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75d5cb9587-9g8g5_14f51478-1e12-478a-91fb-f0e7d69d7447/manager/0.log" Sep 08 17:27:09.328095 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:09.328058 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84f58c54b-62m72_4490dbae-ceb9-4f64-a491-fc2f20ecc243/manager/0.log" Sep 08 17:27:09.348942 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:09.348912 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84f58c54b-62m72_4490dbae-ceb9-4f64-a491-fc2f20ecc243/copy-manifests/0.log" Sep 08 17:27:11.032314 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:11.032230 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-hrmr9_1838d52a-5cb2-4d8d-a9e3-c1d93f64d056/openshift-lws-operator/0.log" Sep 08 17:27:12.027348 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:12.027317 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d8677cc5f-pcp9g_7e359e34-b048-4110-9fc0-149b12c3363a/kube-auth-proxy/0.log" Sep 08 17:27:12.138625 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:12.138600 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-vrwnc_f5cd24d0-2d8f-4639-9fac-d4ae2ba1676d/payload-processing/0.log" Sep 08 17:27:12.159583 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:12.159554 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-858549dccb-6vpw7_ec6cae4f-377a-4717-acb8-63166637113a/router/0.log" Sep 08 17:27:14.103494 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:14.103458 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d8677cc5f-pcp9g_7e359e34-b048-4110-9fc0-149b12c3363a/kube-auth-proxy/0.log" Sep 08 17:27:14.222592 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:14.222561 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-vrwnc_f5cd24d0-2d8f-4639-9fac-d4ae2ba1676d/payload-processing/0.log" Sep 08 17:27:14.241148 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:14.241115 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-858549dccb-6vpw7_ec6cae4f-377a-4717-acb8-63166637113a/router/0.log" Sep 08 17:27:16.412225 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:16.412184 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-64p5d_dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4/kube-storage-version-migrator-operator/1.log" Sep 08 17:27:16.414149 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:16.414092 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-64p5d_dc2b4e2c-8c9e-459a-bdb5-c0065e7334e4/kube-storage-version-migrator-operator/0.log" Sep 08 17:27:17.420742 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:17.420684 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9lwh2_149ee825-8ec8-4b9b-8c8c-c96c5b724b29/kube-multus/0.log" Sep 08 17:27:17.592787 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:17.592738 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5qbl5_6275fc1b-99a7-413c-a0fe-628da6828cd1/kube-multus-additional-cni-plugins/0.log" Sep 08 17:27:17.612606 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:17.612583 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5qbl5_6275fc1b-99a7-413c-a0fe-628da6828cd1/egress-router-binary-copy/0.log" Sep 08 17:27:17.631295 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:17.631267 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5qbl5_6275fc1b-99a7-413c-a0fe-628da6828cd1/cni-plugins/0.log" Sep 08 17:27:17.650368 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:17.650348 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5qbl5_6275fc1b-99a7-413c-a0fe-628da6828cd1/bond-cni-plugin/0.log" Sep 08 17:27:17.675964 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:17.675905 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5qbl5_6275fc1b-99a7-413c-a0fe-628da6828cd1/routeoverride-cni/0.log" Sep 08 17:27:17.697215 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:17.697192 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5qbl5_6275fc1b-99a7-413c-a0fe-628da6828cd1/whereabouts-cni-bincopy/0.log" Sep 08 17:27:17.716635 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:17.716613 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5qbl5_6275fc1b-99a7-413c-a0fe-628da6828cd1/whereabouts-cni/0.log" Sep 08 17:27:18.013474 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:18.013413 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qbb9j_5d001365-79a1-4fec-bcf8-3abf7309f1ff/network-metrics-daemon/0.log" Sep 08 17:27:18.028811 ip-10-0-137-108 kubenswrapper[2565]: I0908 17:27:18.028782 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qbb9j_5d001365-79a1-4fec-bcf8-3abf7309f1ff/kube-rbac-proxy/0.log"