Sep 10 15:59:43.430265 ip-10-0-137-172 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 10 15:59:43.430279 ip-10-0-137-172 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 10 15:59:43.430289 ip-10-0-137-172 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 10 15:59:43.430638 ip-10-0-137-172 systemd[1]: Failed to start Kubernetes Kubelet. Sep 10 15:59:53.679451 ip-10-0-137-172 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 10 15:59:53.679471 ip-10-0-137-172 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 112642310eac470d87cca88757f13075 -- Sep 10 16:01:40.347629 ip-10-0-137-172 systemd[1]: Starting Kubernetes Kubelet... Sep 10 16:01:40.802713 ip-10-0-137-172 kubenswrapper[2541]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 16:01:40.802713 ip-10-0-137-172 kubenswrapper[2541]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 16:01:40.802713 ip-10-0-137-172 kubenswrapper[2541]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 16:01:40.802713 ip-10-0-137-172 kubenswrapper[2541]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 16:01:40.802713 ip-10-0-137-172 kubenswrapper[2541]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 16:01:40.805082 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.804982 2541 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 16:01:40.811276 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811255 2541 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 16:01:40.811276 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811275 2541 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 16:01:40.811276 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811279 2541 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 16:01:40.811387 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811283 2541 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 16:01:40.811387 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811286 2541 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 16:01:40.811387 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811289 2541 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 16:01:40.811387 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811292 2541 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 16:01:40.811387 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811307 2541 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 16:01:40.811387 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811310 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 16:01:40.811387 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811313 2541 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 16:01:40.811387 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811316 2541 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 16:01:40.811387 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811318 2541 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 16:01:40.811387 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811321 2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 16:01:40.811387 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811323 2541 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 16:01:40.811387 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811326 2541 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 16:01:40.811387 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811329 2541 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 16:01:40.811387 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811331 2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 16:01:40.811387 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811334 2541 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 16:01:40.811387 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811336 2541 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 16:01:40.811387 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811339 2541 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 16:01:40.811387 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811342 2541 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 16:01:40.811387 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811345 2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 16:01:40.811387 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811348 2541 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 16:01:40.811858 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811350 2541 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 16:01:40.811858 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811353 2541 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 16:01:40.811858 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811355 2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 16:01:40.811858 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811357 2541 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 16:01:40.811858 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811360 2541 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 16:01:40.811858 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811364 2541 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 16:01:40.811858 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811368 2541 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 16:01:40.811858 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811371 2541 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 16:01:40.811858 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811374 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 16:01:40.811858 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811377 2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 16:01:40.811858 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811380 2541 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 16:01:40.811858 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811382 2541 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 16:01:40.811858 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811387 2541 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 16:01:40.811858 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811391 2541 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 16:01:40.811858 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811394 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 16:01:40.811858 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811397 2541 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 16:01:40.811858 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811400 2541 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 16:01:40.811858 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811402 2541 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 16:01:40.811858 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811405 2541 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 16:01:40.812494 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811408 2541 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 16:01:40.812494 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811411 2541 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 16:01:40.812494 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811413 2541 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 16:01:40.812494 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811417 2541 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 16:01:40.812494 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811419 2541 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 16:01:40.812494 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811421 2541 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 16:01:40.812494 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811424 2541 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 16:01:40.812494 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811428 2541 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 16:01:40.812494 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811432 2541 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 16:01:40.812494 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811435 2541 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 16:01:40.812494 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811438 2541 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 16:01:40.812494 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811440 2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 16:01:40.812494 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811443 2541 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 16:01:40.812494 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811446 2541 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 16:01:40.812494 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811449 2541 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 16:01:40.812494 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811451 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 16:01:40.812494 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811454 2541 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 16:01:40.812494 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811456 2541 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 16:01:40.812494 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811459 2541 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 16:01:40.812494 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811461 2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 16:01:40.813139 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811464 2541 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 16:01:40.813139 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811467 2541 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 16:01:40.813139 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811469 2541 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 16:01:40.813139 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811472 2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 16:01:40.813139 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811474 2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 16:01:40.813139 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811477 2541 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 16:01:40.813139 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811480 2541 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 16:01:40.813139 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811482 2541 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 16:01:40.813139 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811485 2541 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 16:01:40.813139 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811487 2541 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 16:01:40.813139 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811490 2541 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 16:01:40.813139 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811492 2541 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 16:01:40.813139 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811495 2541 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 16:01:40.813139 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811497 2541 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 16:01:40.813139 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811500 2541 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 16:01:40.813139 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811502 2541 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 16:01:40.813139 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811505 2541 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 16:01:40.813139 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811508 2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 16:01:40.813139 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811511 2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 16:01:40.813139 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811513 2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 16:01:40.813650 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811516 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 16:01:40.813650 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811518 2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 16:01:40.813650 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811521 2541 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 16:01:40.813650 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811523 2541 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 16:01:40.813650 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811526 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 16:01:40.813650 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811528 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 16:01:40.813650 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811531 2541 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 16:01:40.813650 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811533 2541 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 16:01:40.813650 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811535 2541 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 16:01:40.813650 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811538 2541 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 16:01:40.813650 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811540 2541 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 16:01:40.813650 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811543 2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 16:01:40.813650 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811545 2541 feature_gate.go:328] unrecognized feature gate: Example Sep 10 16:01:40.813650 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811548 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 16:01:40.813650 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811551 2541 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 16:01:40.813650 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811554 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 16:01:40.813650 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.811557 2541 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 16:01:40.814052 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813667 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 16:01:40.814052 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813680 2541 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 16:01:40.814052 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813684 2541 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 16:01:40.814052 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813688 2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 16:01:40.814052 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813691 2541 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 16:01:40.814052 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813694 2541 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 16:01:40.814052 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813697 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 16:01:40.814052 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813700 2541 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 16:01:40.814052 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813702 2541 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 16:01:40.814052 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813705 2541 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 16:01:40.814052 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813708 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 16:01:40.814052 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813710 2541 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 16:01:40.814052 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813713 2541 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 16:01:40.814052 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813715 2541 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 16:01:40.814052 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813718 2541 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 16:01:40.814052 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813720 2541 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 16:01:40.814052 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813723 2541 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 16:01:40.814052 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813725 2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 16:01:40.814052 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813728 2541 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 16:01:40.814052 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813730 2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 16:01:40.814565 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813733 2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 16:01:40.814565 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813738 2541 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 16:01:40.814565 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813741 2541 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 16:01:40.814565 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813748 2541 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 16:01:40.814565 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813751 2541 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 16:01:40.814565 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813754 2541 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 16:01:40.814565 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813756 2541 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 16:01:40.814565 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813759 2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 16:01:40.814565 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813761 2541 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 16:01:40.814565 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813764 2541 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 16:01:40.814565 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813766 2541 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 16:01:40.814565 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813769 2541 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 16:01:40.814565 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813771 2541 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 16:01:40.814565 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813774 2541 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 16:01:40.814565 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813777 2541 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 16:01:40.814565 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813780 2541 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 16:01:40.814565 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813784 2541 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 16:01:40.814565 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813786 2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 16:01:40.814565 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813789 2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 16:01:40.814565 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813792 2541 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 16:01:40.815053 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813794 2541 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 16:01:40.815053 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813796 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 16:01:40.815053 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813799 2541 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 16:01:40.815053 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813801 2541 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 16:01:40.815053 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813803 2541 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 16:01:40.815053 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813806 2541 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 16:01:40.815053 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813808 2541 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 16:01:40.815053 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813811 2541 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 16:01:40.815053 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813813 2541 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 16:01:40.815053 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813817 2541 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 16:01:40.815053 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813821 2541 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 16:01:40.815053 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813824 2541 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 16:01:40.815053 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813827 2541 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 16:01:40.815053 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813830 2541 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 16:01:40.815053 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813832 2541 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 16:01:40.815053 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813835 2541 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 16:01:40.815053 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813837 2541 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 16:01:40.815053 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813840 2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 16:01:40.815053 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813842 2541 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 16:01:40.815523 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813845 2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 16:01:40.815523 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813847 2541 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 16:01:40.815523 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813850 2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 16:01:40.815523 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813852 2541 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 16:01:40.815523 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813856 2541 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 16:01:40.815523 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813859 2541 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 16:01:40.815523 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813864 2541 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 16:01:40.815523 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813866 2541 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 16:01:40.815523 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813869 2541 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 16:01:40.815523 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813874 2541 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 16:01:40.815523 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813878 2541 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 16:01:40.815523 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813881 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 16:01:40.815523 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813884 2541 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 16:01:40.815523 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813887 2541 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 16:01:40.815523 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813890 2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 16:01:40.815523 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813893 2541 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 16:01:40.815523 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813896 2541 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 16:01:40.815523 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813899 2541 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 16:01:40.815523 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813901 2541 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 16:01:40.815987 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813904 2541 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 16:01:40.815987 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813906 2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 16:01:40.815987 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813909 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 16:01:40.815987 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813911 2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 16:01:40.815987 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813913 2541 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 16:01:40.815987 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813917 2541 feature_gate.go:328] unrecognized feature gate: Example Sep 10 16:01:40.815987 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813919 2541 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 16:01:40.815987 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813922 2541 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 16:01:40.815987 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813924 2541 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 16:01:40.815987 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813927 2541 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 16:01:40.815987 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813929 2541 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 16:01:40.815987 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813932 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 16:01:40.815987 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813934 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 16:01:40.815987 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813937 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 16:01:40.815987 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813939 2541 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 16:01:40.815987 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813942 2541 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 16:01:40.815987 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813944 2541 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 16:01:40.815987 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813946 2541 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 16:01:40.815987 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813949 2541 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 16:01:40.815987 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813952 2541 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 16:01:40.816513 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.813954 2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 16:01:40.816513 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814367 2541 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 16:01:40.816513 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814373 2541 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 16:01:40.816513 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814376 2541 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 16:01:40.816513 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814379 2541 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 16:01:40.816513 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814381 2541 feature_gate.go:328] unrecognized feature gate: Example Sep 10 16:01:40.816513 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814384 2541 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 16:01:40.816513 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814386 2541 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 16:01:40.816513 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814389 2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 16:01:40.816513 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814391 2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 16:01:40.816513 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814394 2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 16:01:40.816513 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814397 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 16:01:40.816513 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814400 2541 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 16:01:40.816513 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814402 2541 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 16:01:40.816513 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814405 2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 16:01:40.816513 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814408 2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 16:01:40.816513 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814410 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 16:01:40.816513 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814414 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 16:01:40.816513 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814417 2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 16:01:40.816513 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814420 2541 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 16:01:40.817001 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814422 2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 16:01:40.817001 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814424 2541 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 16:01:40.817001 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814427 2541 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 16:01:40.817001 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814429 2541 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 16:01:40.817001 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814432 2541 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 16:01:40.817001 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814434 2541 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 16:01:40.817001 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814437 2541 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 16:01:40.817001 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814439 2541 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 16:01:40.817001 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814442 2541 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 16:01:40.817001 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814444 2541 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 16:01:40.817001 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814449 2541 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 16:01:40.817001 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814453 2541 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 16:01:40.817001 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814456 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 16:01:40.817001 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814459 2541 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 16:01:40.817001 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814462 2541 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 16:01:40.817001 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814465 2541 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 16:01:40.817001 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814468 2541 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 16:01:40.817001 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814471 2541 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 16:01:40.817001 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814473 2541 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 16:01:40.817001 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814476 2541 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 16:01:40.817498 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814479 2541 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 16:01:40.817498 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814481 2541 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 16:01:40.817498 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814484 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 16:01:40.817498 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814487 2541 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 16:01:40.817498 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814490 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 16:01:40.817498 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814492 2541 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 16:01:40.817498 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814495 2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 16:01:40.817498 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814497 2541 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 16:01:40.817498 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814500 2541 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 16:01:40.817498 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814502 2541 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 16:01:40.817498 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814507 2541 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 16:01:40.817498 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814509 2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 16:01:40.817498 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814511 2541 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 16:01:40.817498 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814514 2541 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 16:01:40.817498 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814516 2541 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 16:01:40.817498 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814520 2541 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 16:01:40.817498 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814523 2541 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 16:01:40.817498 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814525 2541 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 16:01:40.817498 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814527 2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 16:01:40.817498 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814530 2541 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 16:01:40.818022 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814532 2541 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 16:01:40.818022 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814535 2541 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 16:01:40.818022 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814537 2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 16:01:40.818022 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814540 2541 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 16:01:40.818022 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814542 2541 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 16:01:40.818022 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814545 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 16:01:40.818022 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814547 2541 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 16:01:40.818022 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814550 2541 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 16:01:40.818022 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814552 2541 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 16:01:40.818022 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814555 2541 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 16:01:40.818022 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814557 2541 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 16:01:40.818022 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814560 2541 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 16:01:40.818022 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814563 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 16:01:40.818022 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814565 2541 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 16:01:40.818022 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814567 2541 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 16:01:40.818022 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814570 2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 16:01:40.818022 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814574 2541 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 16:01:40.818022 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814577 2541 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 16:01:40.818022 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814581 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 16:01:40.818497 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814584 2541 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 16:01:40.818497 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814586 2541 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 16:01:40.818497 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814589 2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 16:01:40.818497 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814592 2541 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 16:01:40.818497 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814595 2541 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 16:01:40.818497 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814597 2541 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 16:01:40.818497 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814600 2541 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 16:01:40.818497 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814603 2541 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 16:01:40.818497 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814606 2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 16:01:40.818497 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814608 2541 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 16:01:40.818497 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814611 2541 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 16:01:40.818497 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814613 2541 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 16:01:40.818497 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814616 2541 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 16:01:40.818497 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814618 2541 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 16:01:40.818497 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814621 2541 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 16:01:40.818497 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814628 2541 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 16:01:40.818497 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814630 2541 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 16:01:40.818497 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814632 2541 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 16:01:40.818497 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814635 2541 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 16:01:40.818497 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814639 2541 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.814641 2541 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814722 2541 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814768 2541 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814776 2541 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814781 2541 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814785 2541 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814788 2541 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814792 2541 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814797 2541 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814800 2541 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814803 2541 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814807 2541 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814810 2541 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814813 2541 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814816 2541 flags.go:64] FLAG: --cgroup-root="" Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814819 2541 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814822 2541 flags.go:64] FLAG: --client-ca-file="" Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814825 2541 flags.go:64] FLAG: --cloud-provider="external" Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814828 2541 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814832 2541 flags.go:64] FLAG: --cluster-domain="" Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814835 2541 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814838 2541 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814841 2541 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 16:01:40.818974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814844 2541 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814852 2541 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814855 2541 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814858 2541 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814863 2541 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814866 2541 flags.go:64] FLAG: --contention-profiling="false" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814869 2541 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814871 2541 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814874 2541 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814877 2541 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814881 2541 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814884 2541 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814888 2541 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814891 2541 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814894 2541 flags.go:64] FLAG: --enable-server="true" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814897 2541 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814902 2541 flags.go:64] FLAG: --event-burst="100" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814905 2541 flags.go:64] FLAG: --event-qps="50" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814907 2541 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814910 2541 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814913 2541 flags.go:64] FLAG: --eviction-hard="" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814917 2541 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814920 2541 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814922 2541 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814925 2541 flags.go:64] FLAG: --eviction-soft="" Sep 10 16:01:40.819642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814928 2541 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814930 2541 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814934 2541 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814937 2541 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814940 2541 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814942 2541 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814945 2541 flags.go:64] FLAG: --feature-gates="" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814949 2541 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814952 2541 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814955 2541 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814958 2541 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814962 2541 flags.go:64] FLAG: --healthz-port="10248" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814965 2541 flags.go:64] FLAG: --help="false" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814968 2541 flags.go:64] FLAG: --hostname-override="ip-10-0-137-172.ec2.internal" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814971 2541 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814974 2541 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814976 2541 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814979 2541 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814983 2541 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814986 2541 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814989 2541 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814991 2541 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814994 2541 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 16:01:40.820344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.814997 2541 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815000 2541 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815002 2541 flags.go:64] FLAG: --kube-reserved="" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815005 2541 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815007 2541 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815010 2541 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815013 2541 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815016 2541 flags.go:64] FLAG: --lock-file="" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815018 2541 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815021 2541 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815024 2541 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815029 2541 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815033 2541 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815035 2541 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815038 2541 flags.go:64] FLAG: --logging-format="text" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815040 2541 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815044 2541 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815047 2541 flags.go:64] FLAG: --manifest-url="" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815049 2541 flags.go:64] FLAG: --manifest-url-header="" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815053 2541 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815058 2541 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815062 2541 flags.go:64] FLAG: --max-pods="110" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815065 2541 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815068 2541 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815071 2541 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 16:01:40.820930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815074 2541 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815077 2541 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815080 2541 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815083 2541 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815090 2541 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815093 2541 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815096 2541 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815099 2541 flags.go:64] FLAG: --pod-cidr="" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815102 2541 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815107 2541 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815110 2541 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815113 2541 flags.go:64] FLAG: --pods-per-core="0" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815116 2541 flags.go:64] FLAG: --port="10250" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815119 2541 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815121 2541 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08792e406a21403c9" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815125 2541 flags.go:64] FLAG: --qos-reserved="" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815128 2541 flags.go:64] FLAG: --read-only-port="10255" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815130 2541 flags.go:64] FLAG: --register-node="true" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815133 2541 flags.go:64] FLAG: --register-with-taints="" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815137 2541 flags.go:64] FLAG: --registry-burst="10" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815140 2541 flags.go:64] FLAG: --registry-qps="5" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815142 2541 flags.go:64] FLAG: --reserved-cpus="" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815145 2541 flags.go:64] FLAG: --reserved-memory="" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815148 2541 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815151 2541 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 16:01:40.821541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815154 2541 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815157 2541 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815160 2541 flags.go:64] FLAG: --runonce="false" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815164 2541 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815166 2541 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815169 2541 flags.go:64] FLAG: --seccomp-default="false" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815172 2541 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815175 2541 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815178 2541 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815182 2541 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815185 2541 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815188 2541 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815191 2541 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815194 2541 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815196 2541 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815199 2541 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815202 2541 flags.go:64] FLAG: --system-cgroups="" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815204 2541 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815209 2541 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815212 2541 flags.go:64] FLAG: --tls-cert-file="" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815215 2541 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815219 2541 flags.go:64] FLAG: --tls-min-version="" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815221 2541 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815224 2541 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815227 2541 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 16:01:40.822149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815229 2541 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 16:01:40.822801 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815232 2541 flags.go:64] FLAG: --v="2" Sep 10 16:01:40.822801 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815236 2541 flags.go:64] FLAG: --version="false" Sep 10 16:01:40.822801 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815240 2541 flags.go:64] FLAG: --vmodule="" Sep 10 16:01:40.822801 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815244 2541 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 16:01:40.822801 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.815247 2541 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 16:01:40.822801 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815379 2541 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 16:01:40.822801 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815384 2541 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 16:01:40.822801 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815387 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 16:01:40.822801 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815389 2541 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 16:01:40.822801 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815394 2541 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 16:01:40.822801 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815397 2541 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 16:01:40.822801 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815400 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 16:01:40.822801 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815403 2541 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 16:01:40.822801 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815405 2541 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 16:01:40.822801 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815408 2541 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 16:01:40.822801 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815415 2541 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 16:01:40.822801 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815419 2541 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 16:01:40.822801 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815422 2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 16:01:40.822801 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815424 2541 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 16:01:40.822801 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815427 2541 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 16:01:40.822801 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815430 2541 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 16:01:40.823322 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815433 2541 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 16:01:40.823322 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815436 2541 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 16:01:40.823322 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815438 2541 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 16:01:40.823322 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815440 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 16:01:40.823322 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815443 2541 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 16:01:40.823322 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815445 2541 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 16:01:40.823322 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815448 2541 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 16:01:40.823322 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815467 2541 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 16:01:40.823322 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815471 2541 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 16:01:40.823322 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815474 2541 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 16:01:40.823322 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815477 2541 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 16:01:40.823322 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815479 2541 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 16:01:40.823322 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815482 2541 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 16:01:40.823322 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815484 2541 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 16:01:40.823322 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815490 2541 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 16:01:40.823322 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815492 2541 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 16:01:40.823322 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815495 2541 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 16:01:40.823322 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815497 2541 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 16:01:40.823322 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815500 2541 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 16:01:40.823322 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815502 2541 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 16:01:40.823806 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815507 2541 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 16:01:40.823806 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815510 2541 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 16:01:40.823806 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815512 2541 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 16:01:40.823806 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815515 2541 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 16:01:40.823806 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815518 2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 16:01:40.823806 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815520 2541 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 16:01:40.823806 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815524 2541 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 16:01:40.823806 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815527 2541 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 16:01:40.823806 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815529 2541 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 16:01:40.823806 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815532 2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 16:01:40.823806 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815534 2541 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 16:01:40.823806 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815537 2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 16:01:40.823806 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815539 2541 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 16:01:40.823806 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815542 2541 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 16:01:40.823806 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815545 2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 16:01:40.823806 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815547 2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 16:01:40.823806 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815550 2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 16:01:40.823806 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815552 2541 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 16:01:40.823806 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815555 2541 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 16:01:40.823806 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815557 2541 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 16:01:40.824355 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815560 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 16:01:40.824355 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815563 2541 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 16:01:40.824355 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815565 2541 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 16:01:40.824355 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815568 2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 16:01:40.824355 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815570 2541 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 16:01:40.824355 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815573 2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 16:01:40.824355 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815575 2541 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 16:01:40.824355 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815578 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 16:01:40.824355 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815580 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 16:01:40.824355 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815583 2541 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 16:01:40.824355 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815585 2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 16:01:40.824355 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815588 2541 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 16:01:40.824355 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815591 2541 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 16:01:40.824355 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815593 2541 feature_gate.go:328] unrecognized feature gate: Example Sep 10 16:01:40.824355 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815596 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 16:01:40.824355 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815598 2541 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 16:01:40.824355 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815601 2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 16:01:40.824355 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815603 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 16:01:40.824355 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815608 2541 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 16:01:40.824822 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815612 2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 16:01:40.824822 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815615 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 16:01:40.824822 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815618 2541 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 16:01:40.824822 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815620 2541 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 16:01:40.824822 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815624 2541 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 16:01:40.824822 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815627 2541 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 16:01:40.824822 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815630 2541 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 16:01:40.824822 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815633 2541 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 16:01:40.824822 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815686 2541 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 16:01:40.824822 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815692 2541 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 16:01:40.824822 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815695 2541 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 16:01:40.824822 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815698 2541 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 16:01:40.824822 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815701 2541 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 16:01:40.824822 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815703 2541 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 16:01:40.824822 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815706 2541 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 16:01:40.824822 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815709 2541 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 16:01:40.824822 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815711 2541 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 16:01:40.824822 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815714 2541 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 16:01:40.824822 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815717 2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 16:01:40.825291 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815719 2541 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 16:01:40.825291 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815722 2541 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 16:01:40.825291 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815725 2541 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 16:01:40.825291 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815728 2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 16:01:40.825291 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.815730 2541 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 16:01:40.825291 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.816464 2541 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 16:01:40.825291 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.824503 2541 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 16:01:40.825291 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.824525 2541 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 16:01:40.825518 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.825485 2541 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 16:01:40.825518 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.825493 2541 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 16:01:40.825570 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825553 2541 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 16:01:40.825570 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825559 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 16:01:40.825570 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825563 2541 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 16:01:40.825570 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825566 2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 16:01:40.825570 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825569 2541 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 16:01:40.825570 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825571 2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 16:01:40.825723 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825574 2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 16:01:40.825723 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825577 2541 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 16:01:40.825723 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825580 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 16:01:40.825723 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825582 2541 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 16:01:40.825723 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825585 2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 16:01:40.825723 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825588 2541 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 16:01:40.825723 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825590 2541 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 16:01:40.825723 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825593 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 16:01:40.825723 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825596 2541 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 16:01:40.825723 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825598 2541 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 16:01:40.825723 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825601 2541 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 16:01:40.825723 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825604 2541 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 16:01:40.825723 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825608 2541 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 16:01:40.825723 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825612 2541 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 16:01:40.825723 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825614 2541 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 16:01:40.825723 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825617 2541 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 16:01:40.825723 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825620 2541 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 16:01:40.825723 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825624 2541 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 16:01:40.825723 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825628 2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 16:01:40.826186 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825630 2541 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 16:01:40.826186 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825633 2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 16:01:40.826186 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825635 2541 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 16:01:40.826186 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825637 2541 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 16:01:40.826186 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825641 2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 16:01:40.826186 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825644 2541 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 16:01:40.826186 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825647 2541 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 16:01:40.826186 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825650 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 16:01:40.826186 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825652 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 16:01:40.826186 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825655 2541 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 16:01:40.826186 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825657 2541 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 16:01:40.826186 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825660 2541 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 16:01:40.826186 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825662 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 16:01:40.826186 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825665 2541 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 16:01:40.826186 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825668 2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 16:01:40.826186 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825670 2541 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 16:01:40.826186 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825673 2541 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 16:01:40.826186 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825676 2541 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 16:01:40.826186 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825678 2541 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 16:01:40.826186 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825680 2541 feature_gate.go:328] unrecognized feature gate: Example Sep 10 16:01:40.826686 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825683 2541 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 16:01:40.826686 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825685 2541 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 16:01:40.826686 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825688 2541 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 16:01:40.826686 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825690 2541 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 16:01:40.826686 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825693 2541 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 16:01:40.826686 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825695 2541 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 16:01:40.826686 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825698 2541 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 16:01:40.826686 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825701 2541 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 16:01:40.826686 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825703 2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 16:01:40.826686 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825706 2541 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 16:01:40.826686 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825708 2541 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 16:01:40.826686 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825711 2541 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 16:01:40.826686 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825713 2541 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 16:01:40.826686 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825716 2541 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 16:01:40.826686 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825718 2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 16:01:40.826686 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825720 2541 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 16:01:40.826686 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825724 2541 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 16:01:40.826686 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825727 2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 16:01:40.826686 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825730 2541 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 16:01:40.826686 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825733 2541 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 16:01:40.827172 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825735 2541 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 16:01:40.827172 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825738 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 16:01:40.827172 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825741 2541 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 16:01:40.827172 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825743 2541 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 16:01:40.827172 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825745 2541 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 16:01:40.827172 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825748 2541 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 16:01:40.827172 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825751 2541 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 16:01:40.827172 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825754 2541 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 16:01:40.827172 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825756 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 16:01:40.827172 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825758 2541 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 16:01:40.827172 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825761 2541 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 16:01:40.827172 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825763 2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 16:01:40.827172 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825766 2541 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 16:01:40.827172 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825768 2541 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 16:01:40.827172 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825771 2541 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 16:01:40.827172 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825773 2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 16:01:40.827172 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825776 2541 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 16:01:40.827172 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825778 2541 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 16:01:40.827172 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825781 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 16:01:40.827172 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825783 2541 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 16:01:40.827729 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825785 2541 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 16:01:40.827729 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825788 2541 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 16:01:40.827729 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825791 2541 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 16:01:40.827729 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825793 2541 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 16:01:40.827729 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825795 2541 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 16:01:40.827729 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825798 2541 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 16:01:40.827729 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825800 2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 16:01:40.827729 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825803 2541 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 16:01:40.827729 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825808 2541 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 16:01:40.827729 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825811 2541 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 16:01:40.827729 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825815 2541 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 16:01:40.827729 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825818 2541 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 16:01:40.827729 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825821 2541 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 16:01:40.827729 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825823 2541 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 16:01:40.828069 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.825828 2541 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 16:01:40.828069 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825949 2541 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 16:01:40.828069 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825953 2541 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 16:01:40.828069 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825956 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 16:01:40.828069 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825959 2541 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 16:01:40.828069 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825962 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 16:01:40.828069 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825964 2541 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 16:01:40.828069 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825967 2541 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 16:01:40.828069 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825969 2541 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 16:01:40.828069 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825972 2541 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 16:01:40.828069 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825974 2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 16:01:40.828069 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825977 2541 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 16:01:40.828069 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825980 2541 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 16:01:40.828069 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825982 2541 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 16:01:40.828069 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825985 2541 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 16:01:40.828447 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825987 2541 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 16:01:40.828447 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825990 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 16:01:40.828447 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825992 2541 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 16:01:40.828447 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825995 2541 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 16:01:40.828447 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.825997 2541 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 16:01:40.828447 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826000 2541 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 16:01:40.828447 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826002 2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 16:01:40.828447 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826005 2541 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 16:01:40.828447 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826008 2541 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 16:01:40.828447 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826010 2541 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 16:01:40.828447 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826013 2541 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 16:01:40.828447 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826015 2541 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 16:01:40.828447 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826018 2541 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 16:01:40.828447 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826020 2541 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 16:01:40.828447 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826023 2541 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 16:01:40.828447 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826026 2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 16:01:40.828447 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826028 2541 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 16:01:40.828447 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826031 2541 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 16:01:40.828447 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826033 2541 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 16:01:40.828447 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826036 2541 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 16:01:40.828928 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826038 2541 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 16:01:40.828928 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826042 2541 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 16:01:40.828928 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826045 2541 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 16:01:40.828928 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826048 2541 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 16:01:40.828928 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826050 2541 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 16:01:40.828928 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826053 2541 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 16:01:40.828928 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826055 2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 16:01:40.828928 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826058 2541 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 16:01:40.828928 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826060 2541 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 16:01:40.828928 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826062 2541 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 16:01:40.828928 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826065 2541 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 16:01:40.828928 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826067 2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 16:01:40.828928 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826070 2541 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 16:01:40.828928 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826072 2541 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 16:01:40.828928 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826075 2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 16:01:40.828928 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826077 2541 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 16:01:40.828928 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826080 2541 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 16:01:40.828928 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826083 2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 16:01:40.828928 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826085 2541 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 16:01:40.828928 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826088 2541 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 16:01:40.829429 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826090 2541 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 16:01:40.829429 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826092 2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 16:01:40.829429 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826095 2541 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 16:01:40.829429 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826097 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 16:01:40.829429 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826100 2541 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 16:01:40.829429 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826102 2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 16:01:40.829429 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826105 2541 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 16:01:40.829429 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826108 2541 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 16:01:40.829429 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826111 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 16:01:40.829429 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826113 2541 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 16:01:40.829429 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826115 2541 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 16:01:40.829429 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826118 2541 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 16:01:40.829429 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826120 2541 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 16:01:40.829429 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826123 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 16:01:40.829429 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826126 2541 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 16:01:40.829429 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826129 2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 16:01:40.829429 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826133 2541 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 16:01:40.829429 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826135 2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 16:01:40.829429 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826138 2541 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 16:01:40.829894 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826141 2541 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 16:01:40.829894 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826143 2541 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 16:01:40.829894 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826146 2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 16:01:40.829894 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826149 2541 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 16:01:40.829894 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826152 2541 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 16:01:40.829894 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826155 2541 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 16:01:40.829894 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826158 2541 feature_gate.go:328] unrecognized feature gate: Example Sep 10 16:01:40.829894 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826160 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 16:01:40.829894 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826163 2541 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 16:01:40.829894 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826165 2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 16:01:40.829894 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826168 2541 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 16:01:40.829894 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826170 2541 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 16:01:40.829894 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826172 2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 16:01:40.829894 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826175 2541 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 16:01:40.829894 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826178 2541 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 16:01:40.829894 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826180 2541 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 16:01:40.829894 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826182 2541 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 16:01:40.829894 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826186 2541 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 16:01:40.829894 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826190 2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 16:01:40.830365 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826193 2541 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 16:01:40.830365 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826195 2541 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 16:01:40.830365 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826199 2541 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 16:01:40.830365 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826201 2541 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 16:01:40.830365 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826204 2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 16:01:40.830365 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826207 2541 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 16:01:40.830365 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:40.826209 2541 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 16:01:40.830365 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.826214 2541 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 16:01:40.830365 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.826353 2541 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 16:01:40.830365 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.828797 2541 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 16:01:40.830365 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.829908 2541 server.go:1019] "Starting client certificate rotation" Sep 10 16:01:40.830365 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.830018 2541 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 16:01:40.830365 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.830056 2541 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 16:01:40.832907 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.832893 2541 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 16:01:40.834100 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.834082 2541 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 16:01:40.845545 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.845509 2541 log.go:25] "Validated CRI v1 runtime API" Sep 10 16:01:40.850490 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.850469 2541 log.go:25] "Validated CRI v1 image API" Sep 10 16:01:40.851361 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.851344 2541 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 16:01:40.855745 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.855721 2541 fs.go:135] Filesystem UUIDs: map[6bbc283c-96b8-4601-baed-8ab6eb200064:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e659157a-8e68-4c19-8e9a-dcc977e40c17:/dev/nvme0n1p3] Sep 10 16:01:40.855816 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.855744 2541 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 16:01:40.860894 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.860774 2541 manager.go:216] Machine: {Timestamp:2026-09-10 16:01:40.859384931 +0000 UTC m=+0.382630120 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3259260 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b238dd590596ebceee26ac3954f7e SystemUUID:ec2b238d-d590-596e-bcee-e26ac3954f7e BootID:11264231-0eac-470d-87cc-a88757f13075 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6e:87:8c:81:6b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6e:87:8c:81:6b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:82:06:a8:4a:86 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 16:01:40.860894 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.860887 2541 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 16:01:40.861006 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.860976 2541 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 16:01:40.862215 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.862198 2541 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 16:01:40.864687 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.864650 2541 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 16:01:40.864836 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.864693 2541 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-172.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 16:01:40.866206 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.866193 2541 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 16:01:40.866249 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.866209 2541 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 16:01:40.866249 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.866219 2541 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 16:01:40.866967 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.866954 2541 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 16:01:40.868923 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.868909 2541 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 16:01:40.873725 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.873708 2541 state_mem.go:36] "Initialized new in-memory state store" Sep 10 16:01:40.873849 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.873839 2541 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 16:01:40.873886 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.873873 2541 kubelet.go:486] "Attempting to sync node with API server" Sep 10 16:01:40.873914 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.873888 2541 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 16:01:40.873914 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.873903 2541 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 16:01:40.873976 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.873915 2541 kubelet.go:398] "Adding apiserver pod source" Sep 10 16:01:40.873976 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.873925 2541 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 16:01:40.874457 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.874428 2541 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 16:01:40.874457 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.874458 2541 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 16:01:40.877031 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.877013 2541 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 16:01:40.880440 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.880425 2541 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 16:01:40.880501 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.880457 2541 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 16:01:40.880679 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.880665 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 16:01:40.880734 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.880687 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 16:01:40.880734 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.880694 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 16:01:40.880734 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.880699 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 16:01:40.880734 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.880707 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 16:01:40.880734 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.880714 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 16:01:40.881326 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.881315 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 16:01:40.881361 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.881328 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 16:01:40.881361 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.881337 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 16:01:40.881361 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.881345 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 16:01:40.881434 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.881367 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 16:01:40.881434 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.881378 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 16:01:40.883221 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.883206 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 16:01:40.883221 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.883222 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 16:01:40.886126 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:40.886095 2541 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 16:01:40.886126 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:40.886121 2541 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 16:01:40.886244 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.886126 2541 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-172.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 16:01:40.887696 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.887683 2541 server.go:1268] "Started kubelet" Sep 10 16:01:40.887804 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.887778 2541 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 16:01:40.887866 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.887783 2541 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 16:01:40.887910 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.887863 2541 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 16:01:40.888582 ip-10-0-137-172 systemd[1]: Started Kubernetes Kubelet. Sep 10 16:01:40.889376 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.889286 2541 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 16:01:40.893164 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:40.893133 2541 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 16:01:40.893643 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.893630 2541 server.go:310] "Adding debug handlers to kubelet server" Sep 10 16:01:40.897717 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.897692 2541 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 16:01:40.898419 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.898398 2541 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 16:01:40.898574 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:40.898549 2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found" Sep 10 16:01:40.899147 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.899129 2541 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 16:01:40.899147 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.899138 2541 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 16:01:40.899365 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.899154 2541 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 16:01:40.899365 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.899270 2541 reconstruct.go:97] "Volume reconstruction finished" Sep 10 16:01:40.899365 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.899278 2541 reconciler.go:29] "Reconciler: start to sync state" Sep 10 16:01:40.899802 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.899631 2541 factory.go:55] Registering systemd factory Sep 10 16:01:40.899802 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.899675 2541 factory.go:223] Registration of the systemd container factory successfully Sep 10 16:01:40.900415 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.900399 2541 factory.go:153] Registering CRI-O factory Sep 10 16:01:40.900479 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.900419 2541 factory.go:223] Registration of the crio container factory successfully Sep 10 16:01:40.900479 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.900470 2541 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 16:01:40.900535 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.900492 2541 factory.go:103] Registering Raw factory Sep 10 16:01:40.900535 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.900507 2541 manager.go:1196] Started watching for new ooms in manager Sep 10 16:01:40.900892 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.900876 2541 manager.go:349] Starting recovery of all containers Sep 10 16:01:40.906921 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:40.906888 2541 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 16:01:40.907077 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:40.906976 2541 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 16:01:40.908245 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:40.907018 2541 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-172.ec2.internal.18d4010b80a76516 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-172.ec2.internal,UID:ip-10-0-137-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-172.ec2.internal,},FirstTimestamp:2026-09-10 16:01:40.887651606 +0000 UTC m=+0.410896794,LastTimestamp:2026-09-10 16:01:40.887651606 +0000 UTC m=+0.410896794,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-172.ec2.internal,}" Sep 10 16:01:40.912594 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.912575 2541 manager.go:354] Recovery completed Sep 10 16:01:40.915701 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.915681 2541 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v6dbn" Sep 10 16:01:40.916878 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.916866 2541 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 16:01:40.919141 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.919122 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 16:01:40.919233 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.919160 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 16:01:40.919233 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.919176 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 16:01:40.919747 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.919730 2541 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 16:01:40.919747 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.919742 2541 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 16:01:40.919747 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.919747 2541 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 16:01:40.919941 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.919761 2541 state_mem.go:36] "Initialized new in-memory state store" Sep 10 16:01:40.920504 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.920489 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 16:01:40.920582 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.920517 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 16:01:40.920582 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.920535 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 16:01:40.921706 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:40.921638 2541 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-172.ec2.internal.18d4010b8287e4eb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-172.ec2.internal,UID:ip-10-0-137-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-172.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-172.ec2.internal,},FirstTimestamp:2026-09-10 16:01:40.919141611 +0000 UTC m=+0.442386803,LastTimestamp:2026-09-10 16:01:40.919141611 +0000 UTC m=+0.442386803,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-172.ec2.internal,}" Sep 10 16:01:40.921990 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.921978 2541 policy_none.go:49] "None policy: Start" Sep 10 16:01:40.922037 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.921995 2541 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 16:01:40.922037 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.922005 2541 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 16:01:40.924016 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.924005 2541 policy_none.go:47] "Start" Sep 10 16:01:40.927097 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.927078 2541 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v6dbn" Sep 10 16:01:40.928677 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:40.928587 2541 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-172.ec2.internal.18d4010b828849db default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-172.ec2.internal,UID:ip-10-0-137-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-172.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-172.ec2.internal,},FirstTimestamp:2026-09-10 16:01:40.919167451 +0000 UTC m=+0.442412645,LastTimestamp:2026-09-10 16:01:40.919167451 +0000 UTC m=+0.442412645,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-172.ec2.internal,}" Sep 10 16:01:40.930446 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:40.930382 2541 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-172.ec2.internal.18d4010b82887f66 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-172.ec2.internal,UID:ip-10-0-137-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-172.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-172.ec2.internal,},FirstTimestamp:2026-09-10 16:01:40.919181158 +0000 UTC m=+0.442426351,LastTimestamp:2026-09-10 16:01:40.919181158 +0000 UTC m=+0.442426351,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-172.ec2.internal,}" Sep 10 16:01:40.964003 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.963967 2541 manager.go:337] "Starting Device Plugin manager" Sep 10 16:01:40.979972 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:40.964025 2541 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 16:01:40.979972 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.964040 2541 server.go:87] "Starting device plugin registration server" Sep 10 16:01:40.979972 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.964385 2541 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 16:01:40.979972 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.964397 2541 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 16:01:40.979972 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.964526 2541 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 16:01:40.979972 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.964612 2541 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 16:01:40.979972 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.964621 2541 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 16:01:40.979972 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:40.965198 2541 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 16:01:40.979972 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:40.965255 2541 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-172.ec2.internal\" not found" Sep 10 16:01:40.979972 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.971319 2541 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 16:01:40.979972 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.972654 2541 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 16:01:40.979972 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.972708 2541 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 16:01:40.979972 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.972731 2541 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 16:01:40.979972 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:40.972777 2541 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 16:01:40.979972 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:40.975088 2541 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 16:01:41.065548 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.065451 2541 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 16:01:41.070293 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.070274 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 16:01:41.070411 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.070334 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 16:01:41.070411 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.070349 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 16:01:41.070411 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.070375 2541 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-172.ec2.internal" Sep 10 16:01:41.073621 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.073584 2541 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal"] Sep 10 16:01:41.073679 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.073672 2541 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 16:01:41.075395 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.075377 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 16:01:41.075487 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.075407 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 16:01:41.075487 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.075422 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 16:01:41.077819 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.077803 2541 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 16:01:41.077941 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.077923 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" Sep 10 16:01:41.077988 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.077974 2541 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 16:01:41.079600 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.079583 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 16:01:41.079668 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.079619 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 16:01:41.079668 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.079630 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 16:01:41.079668 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.079654 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 16:01:41.079668 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.079665 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 16:01:41.079812 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.079635 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 16:01:41.081278 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.081261 2541 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-172.ec2.internal" Sep 10 16:01:41.081376 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:41.081282 2541 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-172.ec2.internal\": node \"ip-10-0-137-172.ec2.internal\" not found" Sep 10 16:01:41.081990 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.081972 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal" Sep 10 16:01:41.082069 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.082010 2541 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 16:01:41.083121 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.083106 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 16:01:41.083175 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.083136 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 16:01:41.083175 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.083147 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 16:01:41.094980 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:41.094794 2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found" Sep 10 16:01:41.101128 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.101106 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/93402cb6c1e59df178a12145f97bf120-config\") pod \"kube-apiserver-proxy-ip-10-0-137-172.ec2.internal\" (UID: \"93402cb6c1e59df178a12145f97bf120\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal" Sep 10 16:01:41.101249 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.101133 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60e762d75cf3fcfb77ba57171376954a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal\" (UID: \"60e762d75cf3fcfb77ba57171376954a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" Sep 10 16:01:41.101249 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.101152 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60e762d75cf3fcfb77ba57171376954a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal\" (UID: \"60e762d75cf3fcfb77ba57171376954a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" Sep 10 16:01:41.107000 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:41.106969 2541 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-172.ec2.internal\" not found" node="ip-10-0-137-172.ec2.internal" Sep 10 16:01:41.111326 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:41.111289 2541 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-172.ec2.internal\" not found" node="ip-10-0-137-172.ec2.internal" Sep 10 16:01:41.195548 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:41.195505 2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found" Sep 10 16:01:41.201932 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.201906 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60e762d75cf3fcfb77ba57171376954a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal\" (UID: \"60e762d75cf3fcfb77ba57171376954a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" Sep 10 16:01:41.202015 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.201938 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60e762d75cf3fcfb77ba57171376954a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal\" (UID: \"60e762d75cf3fcfb77ba57171376954a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" Sep 10 16:01:41.202015 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.201958 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/93402cb6c1e59df178a12145f97bf120-config\") pod \"kube-apiserver-proxy-ip-10-0-137-172.ec2.internal\" (UID: \"93402cb6c1e59df178a12145f97bf120\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal" Sep 10 16:01:41.202079 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.202029 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/93402cb6c1e59df178a12145f97bf120-config\") pod \"kube-apiserver-proxy-ip-10-0-137-172.ec2.internal\" (UID: \"93402cb6c1e59df178a12145f97bf120\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal" Sep 10 16:01:41.202079 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.202031 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60e762d75cf3fcfb77ba57171376954a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal\" (UID: \"60e762d75cf3fcfb77ba57171376954a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" Sep 10 16:01:41.202079 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.202028 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60e762d75cf3fcfb77ba57171376954a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal\" (UID: \"60e762d75cf3fcfb77ba57171376954a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" Sep 10 16:01:41.295667 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:41.295628 2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found" Sep 10 16:01:41.396586 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:41.396494 2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found" Sep 10 16:01:41.410981 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.410955 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" Sep 10 16:01:41.415091 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.415074 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal" Sep 10 16:01:41.497539 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:41.497500 2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found" Sep 10 16:01:41.598064 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:41.598026 2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found" Sep 10 16:01:41.698630 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:41.698555 2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found" Sep 10 16:01:41.799180 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:41.799137 2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found" Sep 10 16:01:41.829536 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.829508 2541 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 16:01:41.829967 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.829703 2541 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 16:01:41.900094 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:41.900062 2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found" Sep 10 16:01:41.930790 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.930749 2541 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 15:56:41 +0000 UTC" deadline="2028-04-16 07:01:41.717820057 +0000 UTC" Sep 10 16:01:41.930790 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:41.930782 2541 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14006h59m59.787041374s" Sep 10 16:01:42.000182 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:42.000112 2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found" Sep 10 16:01:42.026387 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.026358 2541 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 16:01:42.027873 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.027857 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientMemory" Sep 10 16:01:42.027988 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.027892 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 16:01:42.027988 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.027903 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientPID" Sep 10 16:01:42.100321 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:42.100281 2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found" Sep 10 16:01:42.147270 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.147241 2541 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 16:01:42.183852 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:42.183820 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e762d75cf3fcfb77ba57171376954a.slice/crio-c746a0813e95b5f259635627c354b1471995efe1b225ca67ccd8f7d02625632f WatchSource:0}: Error finding container c746a0813e95b5f259635627c354b1471995efe1b225ca67ccd8f7d02625632f: Status 404 returned error can't find the container with id c746a0813e95b5f259635627c354b1471995efe1b225ca67ccd8f7d02625632f Sep 10 16:01:42.184099 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:42.184081 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93402cb6c1e59df178a12145f97bf120.slice/crio-3ea39e5081e1c7d49ddaea00f2d081e6130a6e3b4084632c8e0f4047ea796426 WatchSource:0}: Error finding container 3ea39e5081e1c7d49ddaea00f2d081e6130a6e3b4084632c8e0f4047ea796426: Status 404 returned error can't find the container with id 3ea39e5081e1c7d49ddaea00f2d081e6130a6e3b4084632c8e0f4047ea796426 Sep 10 16:01:42.187875 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.187850 2541 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 16:01:42.199686 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.199667 2541 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" Sep 10 16:01:42.213955 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.213925 2541 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 16:01:42.215990 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.215973 2541 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal" Sep 10 16:01:42.226989 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.226972 2541 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 16:01:42.272957 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.272884 2541 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 16:01:42.308763 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.308738 2541 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 16:01:42.875125 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.875070 2541 apiserver.go:52] "Watching apiserver" Sep 10 16:01:42.889209 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.889172 2541 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 16:01:42.889578 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.889552 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-h5frk","openshift-network-diagnostics/network-check-target-j2m6r","openshift-ovn-kubernetes/ovnkube-node-r4c6j","kube-system/global-pull-secret-syncer-cjj7d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc","openshift-cluster-node-tuning-operator/tuned-7gfgn","openshift-multus/multus-additional-cni-plugins-l9crl","openshift-network-operator/iptables-alerter-lqg46","kube-system/konnectivity-agent-48lj8","kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal","openshift-image-registry/node-ca-rxmz9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal","openshift-multus/multus-wjm74"] Sep 10 16:01:42.892287 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.892259 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lqg46" Sep 10 16:01:42.894747 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.894723 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:01:42.894747 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.894745 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-z9t9p\"" Sep 10 16:01:42.894932 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:42.894806 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2m6r" podUID="ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e" Sep 10 16:01:42.894932 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.894846 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 16:01:42.895189 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.895103 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 16:01:42.895335 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.895106 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 16:01:42.897854 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.897827 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wjm74" Sep 10 16:01:42.898461 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.898441 2541 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 16:01:42.899871 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.899806 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 16:01:42.900096 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.900078 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 16:01:42.900223 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.900197 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 16:01:42.900334 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.900080 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6hhzh\"" Sep 10 16:01:42.900506 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.900488 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 16:01:42.901962 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.901659 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-48lj8" Sep 10 16:01:42.903353 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.903322 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 16:01:42.903679 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.903661 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wfqlt\"" Sep 10 16:01:42.903860 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.903842 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 16:01:42.904265 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.904245 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:42.906154 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.906135 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 16:01:42.906252 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.906180 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 16:01:42.906422 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.906403 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fmw4p\"" Sep 10 16:01:42.906733 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.906709 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:42.908457 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.908439 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lg7rc\"" Sep 10 16:01:42.908786 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.908768 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 16:01:42.908786 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.908782 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 16:01:42.908922 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.908826 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:01:42.908922 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:42.908894 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5frk" podUID="086a5470-5fac-4be6-963c-7b219b25b740" Sep 10 16:01:42.911008 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.910832 2541 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 16:01:42.911655 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.911634 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-etc-kubernetes\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:42.911751 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.911670 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-cnibin\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:42.911751 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.911698 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-multus-socket-dir-parent\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:42.911751 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.911724 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-host-var-lib-kubelet\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:42.911919 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.911776 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40f27517-24fa-41df-8a4d-1eefd387cf5d-agent-certs\") pod \"konnectivity-agent-48lj8\" (UID: \"40f27517-24fa-41df-8a4d-1eefd387cf5d\") " pod="kube-system/konnectivity-agent-48lj8" Sep 10 16:01:42.911919 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.911813 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67781cee-0ff9-4fb7-9e88-056433782dd5-iptables-alerter-script\") pod \"iptables-alerter-lqg46\" (UID: \"67781cee-0ff9-4fb7-9e88-056433782dd5\") " pod="openshift-network-operator/iptables-alerter-lqg46" Sep 10 16:01:42.911919 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.911844 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkj84\" (UniqueName: \"kubernetes.io/projected/67781cee-0ff9-4fb7-9e88-056433782dd5-kube-api-access-mkj84\") pod \"iptables-alerter-lqg46\" (UID: \"67781cee-0ff9-4fb7-9e88-056433782dd5\") " pod="openshift-network-operator/iptables-alerter-lqg46" Sep 10 16:01:42.911919 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.911867 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e4613e9-e2c0-4975-90e5-ce76c0e39814-cni-binary-copy\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:42.911919 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.911895 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-host-var-lib-cni-bin\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:42.911919 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.911919 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-multus-conf-dir\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:42.912215 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.911941 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:42.912215 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.911942 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2e4613e9-e2c0-4975-90e5-ce76c0e39814-multus-daemon-config\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:42.912215 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.912077 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55qt4\" (UniqueName: \"kubernetes.io/projected/2e4613e9-e2c0-4975-90e5-ce76c0e39814-kube-api-access-55qt4\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:42.912215 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.912104 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40f27517-24fa-41df-8a4d-1eefd387cf5d-konnectivity-ca\") pod \"konnectivity-agent-48lj8\" (UID: \"40f27517-24fa-41df-8a4d-1eefd387cf5d\") " pod="kube-system/konnectivity-agent-48lj8" Sep 10 16:01:42.912215 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.912156 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67781cee-0ff9-4fb7-9e88-056433782dd5-host-slash\") pod \"iptables-alerter-lqg46\" (UID: \"67781cee-0ff9-4fb7-9e88-056433782dd5\") " pod="openshift-network-operator/iptables-alerter-lqg46" Sep 10 16:01:42.912215 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.912182 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhqbq\" (UniqueName: \"kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq\") pod \"network-check-target-j2m6r\" (UID: \"ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e\") " pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:01:42.912215 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.912206 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-multus-cni-dir\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:42.912588 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.912229 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-host-run-k8s-cni-cncf-io\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:42.912588 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.912250 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-system-cni-dir\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:42.912588 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.912293 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-host-run-netns\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:42.912588 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.912339 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-host-var-lib-cni-multus\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:42.912588 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.912365 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-hostroot\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:42.912588 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.912405 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-host-run-multus-certs\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:42.914421 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.913831 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 16:01:42.914421 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.914236 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:01:42.914421 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:42.914299 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjj7d" podUID="e3074957-6ddd-4ded-9ddf-7c2fe68929f0" Sep 10 16:01:42.914622 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.914554 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-688sw\"" Sep 10 16:01:42.915224 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.914816 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 16:01:42.915224 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.914998 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 16:01:42.917027 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.916556 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rxmz9" Sep 10 16:01:42.918167 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.918144 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 16:01:42.918464 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.918445 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nt2cf\"" Sep 10 16:01:42.918545 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.918517 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 16:01:42.918738 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.918718 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 16:01:42.919237 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.919203 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:42.921081 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.921055 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 16:01:42.921081 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.921072 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dk7dz\"" Sep 10 16:01:42.921298 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.921283 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 16:01:42.921394 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.921382 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 16:01:42.921614 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.921482 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 16:01:42.921695 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.921633 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 16:01:42.922290 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.922271 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 16:01:42.940056 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.939999 2541 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x4kj6" Sep 10 16:01:42.948847 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.948823 2541 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x4kj6" Sep 10 16:01:42.978264 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.978195 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal" event={"ID":"93402cb6c1e59df178a12145f97bf120","Type":"ContainerStarted","Data":"3ea39e5081e1c7d49ddaea00f2d081e6130a6e3b4084632c8e0f4047ea796426"} Sep 10 16:01:42.980198 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:42.980176 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" event={"ID":"60e762d75cf3fcfb77ba57171376954a","Type":"ContainerStarted","Data":"c746a0813e95b5f259635627c354b1471995efe1b225ca67ccd8f7d02625632f"} Sep 10 16:01:43.001514 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.000517 2541 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 16:01:43.012598 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.012567 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pfbw\" (UniqueName: \"kubernetes.io/projected/d67875b1-0368-48c9-92ac-daeb4f6f9b03-kube-api-access-5pfbw\") pod \"aws-ebs-csi-driver-node-ct9xc\" (UID: \"d67875b1-0368-48c9-92ac-daeb4f6f9b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:43.012739 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.012611 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-run-openvswitch\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.012739 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.012643 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54359979-f90b-4674-b39d-122f7c847055-system-cni-dir\") pod \"multus-additional-cni-plugins-l9crl\" (UID: \"54359979-f90b-4674-b39d-122f7c847055\") " pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:43.012739 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.012675 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-system-cni-dir\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.012739 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.012701 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-hostroot\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.012739 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.012730 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40f27517-24fa-41df-8a4d-1eefd387cf5d-agent-certs\") pod \"konnectivity-agent-48lj8\" (UID: \"40f27517-24fa-41df-8a4d-1eefd387cf5d\") " pod="kube-system/konnectivity-agent-48lj8" Sep 10 16:01:43.012919 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.012774 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7bmd\" (UniqueName: \"kubernetes.io/projected/b95766bc-a06f-45f1-a0d0-63892b29d68f-kube-api-access-w7bmd\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.012919 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.012811 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-hostroot\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.012919 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.012812 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d67875b1-0368-48c9-92ac-daeb4f6f9b03-device-dir\") pod \"aws-ebs-csi-driver-node-ct9xc\" (UID: \"d67875b1-0368-48c9-92ac-daeb4f6f9b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:43.012919 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.012863 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-var-lib-openvswitch\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.012919 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.012897 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67781cee-0ff9-4fb7-9e88-056433782dd5-iptables-alerter-script\") pod \"iptables-alerter-lqg46\" (UID: \"67781cee-0ff9-4fb7-9e88-056433782dd5\") " pod="openshift-network-operator/iptables-alerter-lqg46" Sep 10 16:01:43.013084 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.012925 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2e4613e9-e2c0-4975-90e5-ce76c0e39814-multus-daemon-config\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.013214 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.013191 2541 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 16:01:43.015289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.013273 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55qt4\" (UniqueName: \"kubernetes.io/projected/2e4613e9-e2c0-4975-90e5-ce76c0e39814-kube-api-access-55qt4\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.015289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.013334 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-run\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.015289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.013367 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d67875b1-0368-48c9-92ac-daeb4f6f9b03-sys-fs\") pod \"aws-ebs-csi-driver-node-ct9xc\" (UID: \"d67875b1-0368-48c9-92ac-daeb4f6f9b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:43.015289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.013395 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-host-cni-bin\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.015289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.013437 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqbq\" (UniqueName: \"kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq\") pod \"network-check-target-j2m6r\" (UID: \"ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e\") " pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:01:43.015289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.013466 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-multus-cni-dir\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.015289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.013492 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-ovnkube-script-lib\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.015289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.013521 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-host-slash\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.015289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.013546 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-host-var-lib-cni-multus\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.015289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.013576 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-etc-kubernetes\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.015289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.013610 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-host\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.015289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.013651 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-run-ovn\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.015289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.013823 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54359979-f90b-4674-b39d-122f7c847055-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l9crl\" (UID: \"54359979-f90b-4674-b39d-122f7c847055\") " pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:43.015289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.013862 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e4613e9-e2c0-4975-90e5-ce76c0e39814-cni-binary-copy\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.015289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.013893 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n8r6\" (UniqueName: \"kubernetes.io/projected/7e5689cb-d3f5-4d8c-b91c-a0e40aeeeef9-kube-api-access-8n8r6\") pod \"node-ca-rxmz9\" (UID: \"7e5689cb-d3f5-4d8c-b91c-a0e40aeeeef9\") " pod="openshift-image-registry/node-ca-rxmz9" Sep 10 16:01:43.015289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.013921 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67781cee-0ff9-4fb7-9e88-056433782dd5-host-slash\") pod \"iptables-alerter-lqg46\" (UID: \"67781cee-0ff9-4fb7-9e88-056433782dd5\") " pod="openshift-network-operator/iptables-alerter-lqg46" Sep 10 16:01:43.015289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.013948 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-host-run-k8s-cni-cncf-io\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.016110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.013977 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-lib-modules\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.016110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014002 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-etc-openvswitch\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.016110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014028 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-ovnkube-config\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.016110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014053 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-env-overrides\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.016110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014082 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-host-run-netns\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.016110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014111 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-host-run-multus-certs\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.016110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014162 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-multus-cni-dir\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.016110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014175 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-etc-modprobe-d\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.016110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014201 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-host-run-netns\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.016110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014227 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-host-run-ovn-kubernetes\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.016110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014254 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-host-cni-netd\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.016110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014254 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2e4613e9-e2c0-4975-90e5-ce76c0e39814-multus-daemon-config\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.016110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014320 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.016110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014346 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-host-run-k8s-cni-cncf-io\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.016110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014368 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-ovn-node-metrics-cert\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.016110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014409 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-host-run-multus-certs\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.016110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014414 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqn9k\" (UniqueName: \"kubernetes.io/projected/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-kube-api-access-cqn9k\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.016843 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014444 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-system-cni-dir\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.016843 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014459 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-host-run-netns\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.016843 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014473 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-host-var-lib-cni-multus\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.016843 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014489 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/54359979-f90b-4674-b39d-122f7c847055-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l9crl\" (UID: \"54359979-f90b-4674-b39d-122f7c847055\") " pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:43.016843 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014503 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67781cee-0ff9-4fb7-9e88-056433782dd5-host-slash\") pod \"iptables-alerter-lqg46\" (UID: \"67781cee-0ff9-4fb7-9e88-056433782dd5\") " pod="openshift-network-operator/iptables-alerter-lqg46" Sep 10 16:01:43.016843 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014527 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/54359979-f90b-4674-b39d-122f7c847055-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l9crl\" (UID: \"54359979-f90b-4674-b39d-122f7c847055\") " pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:43.016843 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014561 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs\") pod \"network-metrics-daemon-h5frk\" (UID: \"086a5470-5fac-4be6-963c-7b219b25b740\") " pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:01:43.016843 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014595 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkj84\" (UniqueName: \"kubernetes.io/projected/67781cee-0ff9-4fb7-9e88-056433782dd5-kube-api-access-mkj84\") pod \"iptables-alerter-lqg46\" (UID: \"67781cee-0ff9-4fb7-9e88-056433782dd5\") " pod="openshift-network-operator/iptables-alerter-lqg46" Sep 10 16:01:43.016843 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014625 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40f27517-24fa-41df-8a4d-1eefd387cf5d-konnectivity-ca\") pod \"konnectivity-agent-48lj8\" (UID: \"40f27517-24fa-41df-8a4d-1eefd387cf5d\") " pod="kube-system/konnectivity-agent-48lj8" Sep 10 16:01:43.016843 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014652 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e5689cb-d3f5-4d8c-b91c-a0e40aeeeef9-host\") pod \"node-ca-rxmz9\" (UID: \"7e5689cb-d3f5-4d8c-b91c-a0e40aeeeef9\") " pod="openshift-image-registry/node-ca-rxmz9" Sep 10 16:01:43.016843 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014685 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb7x8\" (UniqueName: \"kubernetes.io/projected/086a5470-5fac-4be6-963c-7b219b25b740-kube-api-access-cb7x8\") pod \"network-metrics-daemon-h5frk\" (UID: \"086a5470-5fac-4be6-963c-7b219b25b740\") " pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:01:43.016843 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014712 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-etc-sysconfig\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.016843 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014737 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d67875b1-0368-48c9-92ac-daeb4f6f9b03-registration-dir\") pod \"aws-ebs-csi-driver-node-ct9xc\" (UID: \"d67875b1-0368-48c9-92ac-daeb4f6f9b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:43.016843 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014764 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-kubelet-config\") pod \"global-pull-secret-syncer-cjj7d\" (UID: \"e3074957-6ddd-4ded-9ddf-7c2fe68929f0\") " pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:01:43.016843 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014804 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e5689cb-d3f5-4d8c-b91c-a0e40aeeeef9-serviceca\") pod \"node-ca-rxmz9\" (UID: \"7e5689cb-d3f5-4d8c-b91c-a0e40aeeeef9\") " pod="openshift-image-registry/node-ca-rxmz9" Sep 10 16:01:43.016843 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014831 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-node-log\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.017407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014860 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54359979-f90b-4674-b39d-122f7c847055-cni-binary-copy\") pod \"multus-additional-cni-plugins-l9crl\" (UID: \"54359979-f90b-4674-b39d-122f7c847055\") " pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:43.017407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014886 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t48fq\" (UniqueName: \"kubernetes.io/projected/54359979-f90b-4674-b39d-122f7c847055-kube-api-access-t48fq\") pod \"multus-additional-cni-plugins-l9crl\" (UID: \"54359979-f90b-4674-b39d-122f7c847055\") " pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:43.017407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014934 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-etc-kubernetes\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.017407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014959 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d67875b1-0368-48c9-92ac-daeb4f6f9b03-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ct9xc\" (UID: \"d67875b1-0368-48c9-92ac-daeb4f6f9b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:43.017407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.014988 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d67875b1-0368-48c9-92ac-daeb4f6f9b03-socket-dir\") pod \"aws-ebs-csi-driver-node-ct9xc\" (UID: \"d67875b1-0368-48c9-92ac-daeb4f6f9b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:43.017407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015015 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret\") pod \"global-pull-secret-syncer-cjj7d\" (UID: \"e3074957-6ddd-4ded-9ddf-7c2fe68929f0\") " pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:01:43.017407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015040 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-log-socket\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.017407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015046 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e4613e9-e2c0-4975-90e5-ce76c0e39814-cni-binary-copy\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.017407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015065 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54359979-f90b-4674-b39d-122f7c847055-cnibin\") pod \"multus-additional-cni-plugins-l9crl\" (UID: \"54359979-f90b-4674-b39d-122f7c847055\") " pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:43.017407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015046 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67781cee-0ff9-4fb7-9e88-056433782dd5-iptables-alerter-script\") pod \"iptables-alerter-lqg46\" (UID: \"67781cee-0ff9-4fb7-9e88-056433782dd5\") " pod="openshift-network-operator/iptables-alerter-lqg46" Sep 10 16:01:43.017407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015137 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-etc-kubernetes\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.017407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015187 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-cnibin\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.017407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015226 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-multus-socket-dir-parent\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.017407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015230 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-cnibin\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.017407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015239 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40f27517-24fa-41df-8a4d-1eefd387cf5d-konnectivity-ca\") pod \"konnectivity-agent-48lj8\" (UID: \"40f27517-24fa-41df-8a4d-1eefd387cf5d\") " pod="kube-system/konnectivity-agent-48lj8" Sep 10 16:01:43.017407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015283 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-host-var-lib-kubelet\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.017407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015333 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-multus-socket-dir-parent\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.018103 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015375 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-etc-systemd\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.018103 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015398 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b95766bc-a06f-45f1-a0d0-63892b29d68f-etc-tuned\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.018103 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015451 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d67875b1-0368-48c9-92ac-daeb4f6f9b03-etc-selinux\") pod \"aws-ebs-csi-driver-node-ct9xc\" (UID: \"d67875b1-0368-48c9-92ac-daeb4f6f9b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:43.018103 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015477 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-host-var-lib-kubelet\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.018103 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015507 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-dbus\") pod \"global-pull-secret-syncer-cjj7d\" (UID: \"e3074957-6ddd-4ded-9ddf-7c2fe68929f0\") " pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:01:43.018103 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015531 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-host-kubelet\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.018103 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015559 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-host-var-lib-cni-bin\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.018103 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015583 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-multus-conf-dir\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.018103 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015618 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-host-var-lib-cni-bin\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.018103 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015657 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-etc-sysctl-conf\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.018103 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015694 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e4613e9-e2c0-4975-90e5-ce76c0e39814-multus-conf-dir\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.018103 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015742 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-sys\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.018103 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015780 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-var-lib-kubelet\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.018103 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015807 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b95766bc-a06f-45f1-a0d0-63892b29d68f-tmp\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.018103 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015834 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-systemd-units\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.018103 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015868 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-run-systemd\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.018103 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.015893 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-etc-sysctl-d\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.020205 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:43.020186 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 16:01:43.020317 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:43.020221 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 16:01:43.020317 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:43.020234 2541 projected.go:196] Error preparing data for projected volume kube-api-access-rhqbq for pod openshift-network-diagnostics/network-check-target-j2m6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:01:43.020317 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.020187 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40f27517-24fa-41df-8a4d-1eefd387cf5d-agent-certs\") pod \"konnectivity-agent-48lj8\" (UID: \"40f27517-24fa-41df-8a4d-1eefd387cf5d\") " pod="kube-system/konnectivity-agent-48lj8" Sep 10 16:01:43.020317 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:43.020291 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq podName:ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e nodeName:}" failed. No retries permitted until 2026-09-10 16:01:43.520269222 +0000 UTC m=+3.043514399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rhqbq" (UniqueName: "kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq") pod "network-check-target-j2m6r" (UID: "ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:01:43.022917 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.022874 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55qt4\" (UniqueName: \"kubernetes.io/projected/2e4613e9-e2c0-4975-90e5-ce76c0e39814-kube-api-access-55qt4\") pod \"multus-wjm74\" (UID: \"2e4613e9-e2c0-4975-90e5-ce76c0e39814\") " pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.023964 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.023941 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkj84\" (UniqueName: \"kubernetes.io/projected/67781cee-0ff9-4fb7-9e88-056433782dd5-kube-api-access-mkj84\") pod \"iptables-alerter-lqg46\" (UID: \"67781cee-0ff9-4fb7-9e88-056433782dd5\") " pod="openshift-network-operator/iptables-alerter-lqg46" Sep 10 16:01:43.116822 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.116780 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-etc-systemd\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.116822 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.116830 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b95766bc-a06f-45f1-a0d0-63892b29d68f-etc-tuned\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.117085 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.116857 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d67875b1-0368-48c9-92ac-daeb4f6f9b03-etc-selinux\") pod \"aws-ebs-csi-driver-node-ct9xc\" (UID: \"d67875b1-0368-48c9-92ac-daeb4f6f9b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:43.117085 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.116882 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-dbus\") pod \"global-pull-secret-syncer-cjj7d\" (UID: \"e3074957-6ddd-4ded-9ddf-7c2fe68929f0\") " pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:01:43.117085 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.116897 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-etc-systemd\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.117085 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.116906 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-host-kubelet\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.117085 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.116951 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-host-kubelet\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.117085 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.116967 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-etc-sysctl-conf\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.117085 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.116995 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-sys\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.117085 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117017 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-var-lib-kubelet\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.117085 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117040 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b95766bc-a06f-45f1-a0d0-63892b29d68f-tmp\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.117085 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117071 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d67875b1-0368-48c9-92ac-daeb4f6f9b03-etc-selinux\") pod \"aws-ebs-csi-driver-node-ct9xc\" (UID: \"d67875b1-0368-48c9-92ac-daeb4f6f9b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:43.117085 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117070 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-systemd-units\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.117672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117102 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-run-systemd\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.117672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117110 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-systemd-units\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.117672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117124 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-etc-sysctl-d\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.117672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117143 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pfbw\" (UniqueName: \"kubernetes.io/projected/d67875b1-0368-48c9-92ac-daeb4f6f9b03-kube-api-access-5pfbw\") pod \"aws-ebs-csi-driver-node-ct9xc\" (UID: \"d67875b1-0368-48c9-92ac-daeb4f6f9b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:43.117672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117160 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-run-openvswitch\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.117672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117184 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54359979-f90b-4674-b39d-122f7c847055-system-cni-dir\") pod \"multus-additional-cni-plugins-l9crl\" (UID: \"54359979-f90b-4674-b39d-122f7c847055\") " pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:43.117672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117206 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7bmd\" (UniqueName: \"kubernetes.io/projected/b95766bc-a06f-45f1-a0d0-63892b29d68f-kube-api-access-w7bmd\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.117672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117221 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d67875b1-0368-48c9-92ac-daeb4f6f9b03-device-dir\") pod \"aws-ebs-csi-driver-node-ct9xc\" (UID: \"d67875b1-0368-48c9-92ac-daeb4f6f9b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:43.117672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117236 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-var-lib-openvswitch\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.117672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117250 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-etc-sysctl-conf\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.117672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117255 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-run\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.117672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117287 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-run\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.117672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117291 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d67875b1-0368-48c9-92ac-daeb4f6f9b03-sys-fs\") pod \"aws-ebs-csi-driver-node-ct9xc\" (UID: \"d67875b1-0368-48c9-92ac-daeb4f6f9b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:43.117672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117319 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-host-cni-bin\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.117672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117335 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-dbus\") pod \"global-pull-secret-syncer-cjj7d\" (UID: \"e3074957-6ddd-4ded-9ddf-7c2fe68929f0\") " pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:01:43.117672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117395 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-run-systemd\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.117672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117413 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d67875b1-0368-48c9-92ac-daeb4f6f9b03-sys-fs\") pod \"aws-ebs-csi-driver-node-ct9xc\" (UID: \"d67875b1-0368-48c9-92ac-daeb4f6f9b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:43.118554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117419 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54359979-f90b-4674-b39d-122f7c847055-system-cni-dir\") pod \"multus-additional-cni-plugins-l9crl\" (UID: \"54359979-f90b-4674-b39d-122f7c847055\") " pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:43.118554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117460 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-ovnkube-script-lib\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.118554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117473 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-run-openvswitch\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.118554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117490 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-host-slash\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.118554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117516 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-etc-kubernetes\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.118554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117542 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-host\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.118554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117566 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-run-ovn\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.118554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117604 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54359979-f90b-4674-b39d-122f7c847055-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l9crl\" (UID: \"54359979-f90b-4674-b39d-122f7c847055\") " pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:43.118554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117615 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-etc-sysctl-d\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.118554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117632 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n8r6\" (UniqueName: \"kubernetes.io/projected/7e5689cb-d3f5-4d8c-b91c-a0e40aeeeef9-kube-api-access-8n8r6\") pod \"node-ca-rxmz9\" (UID: \"7e5689cb-d3f5-4d8c-b91c-a0e40aeeeef9\") " pod="openshift-image-registry/node-ca-rxmz9" Sep 10 16:01:43.118554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117676 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d67875b1-0368-48c9-92ac-daeb4f6f9b03-device-dir\") pod \"aws-ebs-csi-driver-node-ct9xc\" (UID: \"d67875b1-0368-48c9-92ac-daeb4f6f9b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:43.118554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117677 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-lib-modules\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.118554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117724 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-sys\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.118554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117729 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-var-lib-openvswitch\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.118554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117772 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-var-lib-kubelet\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.118554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117779 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-etc-openvswitch\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.118554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117807 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-etc-kubernetes\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.119488 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117812 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-ovnkube-config\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.119488 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117814 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-lib-modules\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.119488 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117831 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-host-cni-bin\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.119488 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117837 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-env-overrides\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.119488 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117863 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-etc-openvswitch\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.119488 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117906 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-host\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.119488 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117912 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-host-slash\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.119488 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117942 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-etc-modprobe-d\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.119488 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117958 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-run-ovn\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.119488 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.117976 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-host-run-netns\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.119488 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118004 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-host-run-ovn-kubernetes\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.119488 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118033 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-host-cni-netd\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.119488 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118061 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.119488 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118090 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-ovn-node-metrics-cert\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.119488 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118099 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-etc-modprobe-d\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.119488 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118117 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqn9k\" (UniqueName: \"kubernetes.io/projected/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-kube-api-access-cqn9k\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.119488 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118144 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/54359979-f90b-4674-b39d-122f7c847055-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l9crl\" (UID: \"54359979-f90b-4674-b39d-122f7c847055\") " pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:43.120311 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118174 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/54359979-f90b-4674-b39d-122f7c847055-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l9crl\" (UID: \"54359979-f90b-4674-b39d-122f7c847055\") " pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:43.120311 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118200 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs\") pod \"network-metrics-daemon-h5frk\" (UID: \"086a5470-5fac-4be6-963c-7b219b25b740\") " pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:01:43.120311 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118228 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e5689cb-d3f5-4d8c-b91c-a0e40aeeeef9-host\") pod \"node-ca-rxmz9\" (UID: \"7e5689cb-d3f5-4d8c-b91c-a0e40aeeeef9\") " pod="openshift-image-registry/node-ca-rxmz9" Sep 10 16:01:43.120311 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118277 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54359979-f90b-4674-b39d-122f7c847055-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l9crl\" (UID: \"54359979-f90b-4674-b39d-122f7c847055\") " pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:43.120311 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118287 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb7x8\" (UniqueName: \"kubernetes.io/projected/086a5470-5fac-4be6-963c-7b219b25b740-kube-api-access-cb7x8\") pod \"network-metrics-daemon-h5frk\" (UID: \"086a5470-5fac-4be6-963c-7b219b25b740\") " pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:01:43.120311 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118323 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-etc-sysconfig\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.120311 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118329 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-ovnkube-script-lib\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.120311 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118430 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d67875b1-0368-48c9-92ac-daeb4f6f9b03-registration-dir\") pod \"aws-ebs-csi-driver-node-ct9xc\" (UID: \"d67875b1-0368-48c9-92ac-daeb4f6f9b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:43.120311 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118506 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-host-run-netns\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.120311 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118351 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d67875b1-0368-48c9-92ac-daeb4f6f9b03-registration-dir\") pod \"aws-ebs-csi-driver-node-ct9xc\" (UID: \"d67875b1-0368-48c9-92ac-daeb4f6f9b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:43.120311 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118564 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-host-run-ovn-kubernetes\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.120311 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118569 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-kubelet-config\") pod \"global-pull-secret-syncer-cjj7d\" (UID: \"e3074957-6ddd-4ded-9ddf-7c2fe68929f0\") " pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:01:43.120311 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118609 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-host-cni-netd\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.120311 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118627 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-kubelet-config\") pod \"global-pull-secret-syncer-cjj7d\" (UID: \"e3074957-6ddd-4ded-9ddf-7c2fe68929f0\") " pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:01:43.120311 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118640 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e5689cb-d3f5-4d8c-b91c-a0e40aeeeef9-serviceca\") pod \"node-ca-rxmz9\" (UID: \"7e5689cb-d3f5-4d8c-b91c-a0e40aeeeef9\") " pod="openshift-image-registry/node-ca-rxmz9" Sep 10 16:01:43.120311 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118676 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-node-log\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.120311 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118686 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e5689cb-d3f5-4d8c-b91c-a0e40aeeeef9-host\") pod \"node-ca-rxmz9\" (UID: \"7e5689cb-d3f5-4d8c-b91c-a0e40aeeeef9\") " pod="openshift-image-registry/node-ca-rxmz9" Sep 10 16:01:43.121186 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118702 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54359979-f90b-4674-b39d-122f7c847055-cni-binary-copy\") pod \"multus-additional-cni-plugins-l9crl\" (UID: \"54359979-f90b-4674-b39d-122f7c847055\") " pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:43.121186 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118703 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.121186 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118731 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t48fq\" (UniqueName: \"kubernetes.io/projected/54359979-f90b-4674-b39d-122f7c847055-kube-api-access-t48fq\") pod \"multus-additional-cni-plugins-l9crl\" (UID: \"54359979-f90b-4674-b39d-122f7c847055\") " pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:43.121186 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118765 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d67875b1-0368-48c9-92ac-daeb4f6f9b03-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ct9xc\" (UID: \"d67875b1-0368-48c9-92ac-daeb4f6f9b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:43.121186 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118791 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d67875b1-0368-48c9-92ac-daeb4f6f9b03-socket-dir\") pod \"aws-ebs-csi-driver-node-ct9xc\" (UID: \"d67875b1-0368-48c9-92ac-daeb4f6f9b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:43.121186 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118817 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret\") pod \"global-pull-secret-syncer-cjj7d\" (UID: \"e3074957-6ddd-4ded-9ddf-7c2fe68929f0\") " pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:01:43.121186 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118846 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-log-socket\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.121186 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118871 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54359979-f90b-4674-b39d-122f7c847055-cnibin\") pod \"multus-additional-cni-plugins-l9crl\" (UID: \"54359979-f90b-4674-b39d-122f7c847055\") " pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:43.121186 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.118967 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54359979-f90b-4674-b39d-122f7c847055-cnibin\") pod \"multus-additional-cni-plugins-l9crl\" (UID: \"54359979-f90b-4674-b39d-122f7c847055\") " pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:43.121186 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.119212 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b95766bc-a06f-45f1-a0d0-63892b29d68f-etc-sysconfig\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.121186 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.119312 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-env-overrides\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.121186 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.119440 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-ovnkube-config\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.121186 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:43.119477 2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:01:43.121186 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.119515 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-node-log\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.121186 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:43.119563 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs podName:086a5470-5fac-4be6-963c-7b219b25b740 nodeName:}" failed. No retries permitted until 2026-09-10 16:01:43.619542189 +0000 UTC m=+3.142787390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs") pod "network-metrics-daemon-h5frk" (UID: "086a5470-5fac-4be6-963c-7b219b25b740") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:01:43.121186 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.119477 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d67875b1-0368-48c9-92ac-daeb4f6f9b03-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ct9xc\" (UID: \"d67875b1-0368-48c9-92ac-daeb4f6f9b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:43.121186 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.119769 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e5689cb-d3f5-4d8c-b91c-a0e40aeeeef9-serviceca\") pod \"node-ca-rxmz9\" (UID: \"7e5689cb-d3f5-4d8c-b91c-a0e40aeeeef9\") " pod="openshift-image-registry/node-ca-rxmz9" Sep 10 16:01:43.122044 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.119890 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d67875b1-0368-48c9-92ac-daeb4f6f9b03-socket-dir\") pod \"aws-ebs-csi-driver-node-ct9xc\" (UID: \"d67875b1-0368-48c9-92ac-daeb4f6f9b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:43.122044 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.119951 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-log-socket\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.122044 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:43.119972 2541 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 16:01:43.122044 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:43.120036 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret podName:e3074957-6ddd-4ded-9ddf-7c2fe68929f0 nodeName:}" failed. No retries permitted until 2026-09-10 16:01:43.620019474 +0000 UTC m=+3.143264665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret") pod "global-pull-secret-syncer-cjj7d" (UID: "e3074957-6ddd-4ded-9ddf-7c2fe68929f0") : object "kube-system"/"original-pull-secret" not registered Sep 10 16:01:43.122044 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.120452 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/54359979-f90b-4674-b39d-122f7c847055-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l9crl\" (UID: \"54359979-f90b-4674-b39d-122f7c847055\") " pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:43.122044 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.120501 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b95766bc-a06f-45f1-a0d0-63892b29d68f-etc-tuned\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.122044 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.120528 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b95766bc-a06f-45f1-a0d0-63892b29d68f-tmp\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.122044 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.121112 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/54359979-f90b-4674-b39d-122f7c847055-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l9crl\" (UID: \"54359979-f90b-4674-b39d-122f7c847055\") " pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:43.122044 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.121801 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54359979-f90b-4674-b39d-122f7c847055-cni-binary-copy\") pod \"multus-additional-cni-plugins-l9crl\" (UID: \"54359979-f90b-4674-b39d-122f7c847055\") " pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:43.124496 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.124475 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-ovn-node-metrics-cert\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.130745 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.130718 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb7x8\" (UniqueName: \"kubernetes.io/projected/086a5470-5fac-4be6-963c-7b219b25b740-kube-api-access-cb7x8\") pod \"network-metrics-daemon-h5frk\" (UID: \"086a5470-5fac-4be6-963c-7b219b25b740\") " pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:01:43.130885 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.130862 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7bmd\" (UniqueName: \"kubernetes.io/projected/b95766bc-a06f-45f1-a0d0-63892b29d68f-kube-api-access-w7bmd\") pod \"tuned-7gfgn\" (UID: \"b95766bc-a06f-45f1-a0d0-63892b29d68f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.131219 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.131201 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t48fq\" (UniqueName: \"kubernetes.io/projected/54359979-f90b-4674-b39d-122f7c847055-kube-api-access-t48fq\") pod \"multus-additional-cni-plugins-l9crl\" (UID: \"54359979-f90b-4674-b39d-122f7c847055\") " pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:43.132307 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.132271 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqn9k\" (UniqueName: \"kubernetes.io/projected/2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd-kube-api-access-cqn9k\") pod \"ovnkube-node-r4c6j\" (UID: \"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.132582 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.132563 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pfbw\" (UniqueName: \"kubernetes.io/projected/d67875b1-0368-48c9-92ac-daeb4f6f9b03-kube-api-access-5pfbw\") pod \"aws-ebs-csi-driver-node-ct9xc\" (UID: \"d67875b1-0368-48c9-92ac-daeb4f6f9b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:43.132742 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.132719 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n8r6\" (UniqueName: \"kubernetes.io/projected/7e5689cb-d3f5-4d8c-b91c-a0e40aeeeef9-kube-api-access-8n8r6\") pod \"node-ca-rxmz9\" (UID: \"7e5689cb-d3f5-4d8c-b91c-a0e40aeeeef9\") " pod="openshift-image-registry/node-ca-rxmz9" Sep 10 16:01:43.207831 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.207727 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lqg46" Sep 10 16:01:43.213651 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.213620 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wjm74" Sep 10 16:01:43.226541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.226512 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" Sep 10 16:01:43.233523 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.233498 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-48lj8" Sep 10 16:01:43.239501 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.239471 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l9crl" Sep 10 16:01:43.248512 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.248490 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" Sep 10 16:01:43.259445 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.259421 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rxmz9" Sep 10 16:01:43.265743 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.265573 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:01:43.379745 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.379698 2541 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 16:01:43.522339 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.522239 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqbq\" (UniqueName: \"kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq\") pod \"network-check-target-j2m6r\" (UID: \"ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e\") " pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:01:43.522502 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:43.522418 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 16:01:43.522502 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:43.522441 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 16:01:43.522502 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:43.522455 2541 projected.go:196] Error preparing data for projected volume kube-api-access-rhqbq for pod openshift-network-diagnostics/network-check-target-j2m6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:01:43.522674 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:43.522523 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq podName:ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e nodeName:}" failed. No retries permitted until 2026-09-10 16:01:44.522503659 +0000 UTC m=+4.045748836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rhqbq" (UniqueName: "kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq") pod "network-check-target-j2m6r" (UID: "ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:01:43.622920 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.622883 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs\") pod \"network-metrics-daemon-h5frk\" (UID: \"086a5470-5fac-4be6-963c-7b219b25b740\") " pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:01:43.623088 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.622934 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret\") pod \"global-pull-secret-syncer-cjj7d\" (UID: \"e3074957-6ddd-4ded-9ddf-7c2fe68929f0\") " pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:01:43.623088 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:43.623067 2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:01:43.623210 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:43.623141 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs podName:086a5470-5fac-4be6-963c-7b219b25b740 nodeName:}" failed. No retries permitted until 2026-09-10 16:01:44.623123153 +0000 UTC m=+4.146368334 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs") pod "network-metrics-daemon-h5frk" (UID: "086a5470-5fac-4be6-963c-7b219b25b740") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:01:43.623210 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:43.623067 2541 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 16:01:43.623210 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:43.623210 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret podName:e3074957-6ddd-4ded-9ddf-7c2fe68929f0 nodeName:}" failed. No retries permitted until 2026-09-10 16:01:44.623197775 +0000 UTC m=+4.146442952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret") pod "global-pull-secret-syncer-cjj7d" (UID: "e3074957-6ddd-4ded-9ddf-7c2fe68929f0") : object "kube-system"/"original-pull-secret" not registered Sep 10 16:01:43.953538 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.953426 2541 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 15:56:43 +0000 UTC" deadline="2028-06-12 17:18:39.638379558 +0000 UTC" Sep 10 16:01:43.953538 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.953473 2541 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15385h16m55.684912157s" Sep 10 16:01:43.974000 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:43.973965 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:01:43.974179 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:43.974106 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5frk" podUID="086a5470-5fac-4be6-963c-7b219b25b740" Sep 10 16:01:44.243782 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:44.243700 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95766bc_a06f_45f1_a0d0_63892b29d68f.slice/crio-e3e89177876a2c30698e4761245c90a058aec1275f997069c6ddf0b942becfc1 WatchSource:0}: Error finding container e3e89177876a2c30698e4761245c90a058aec1275f997069c6ddf0b942becfc1: Status 404 returned error can't find the container with id e3e89177876a2c30698e4761245c90a058aec1275f997069c6ddf0b942becfc1 Sep 10 16:01:44.246682 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:44.246651 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5689cb_d3f5_4d8c_b91c_a0e40aeeeef9.slice/crio-1735a89c1eebdde95a8beeb6daf5c19fb4a9103c81be1cf52721fc3a1f3aad60 WatchSource:0}: Error finding container 1735a89c1eebdde95a8beeb6daf5c19fb4a9103c81be1cf52721fc3a1f3aad60: Status 404 returned error can't find the container with id 1735a89c1eebdde95a8beeb6daf5c19fb4a9103c81be1cf52721fc3a1f3aad60 Sep 10 16:01:44.247452 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:44.247395 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7c8cbd_bcf1_4ef4_8091_8d9333f251fd.slice/crio-78c58a75d3457ed04a2d5ef76b3e36d911ec6cfc91ff78ff2e6f8ad4a4fc14d8 WatchSource:0}: Error finding container 78c58a75d3457ed04a2d5ef76b3e36d911ec6cfc91ff78ff2e6f8ad4a4fc14d8: Status 404 returned error can't find the container with id 78c58a75d3457ed04a2d5ef76b3e36d911ec6cfc91ff78ff2e6f8ad4a4fc14d8 Sep 10 16:01:44.249002 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:44.248946 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f27517_24fa_41df_8a4d_1eefd387cf5d.slice/crio-1467096ee3353dee2c3fcb1c84baa5207c3f48f867e63fa9326caa093db0410c WatchSource:0}: Error finding container 1467096ee3353dee2c3fcb1c84baa5207c3f48f867e63fa9326caa093db0410c: Status 404 returned error can't find the container with id 1467096ee3353dee2c3fcb1c84baa5207c3f48f867e63fa9326caa093db0410c Sep 10 16:01:44.250675 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:44.250230 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67875b1_0368_48c9_92ac_daeb4f6f9b03.slice/crio-cb8052780f677b4d9cd7ac9c110f9b47d7abc0305949bbbb6892662135cb365f WatchSource:0}: Error finding container cb8052780f677b4d9cd7ac9c110f9b47d7abc0305949bbbb6892662135cb365f: Status 404 returned error can't find the container with id cb8052780f677b4d9cd7ac9c110f9b47d7abc0305949bbbb6892662135cb365f Sep 10 16:01:44.252758 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:44.252500 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67781cee_0ff9_4fb7_9e88_056433782dd5.slice/crio-d5e6404860bc00d728f227b500e1b671a37db209b5bac79dea084ec7f07aef69 WatchSource:0}: Error finding container d5e6404860bc00d728f227b500e1b671a37db209b5bac79dea084ec7f07aef69: Status 404 returned error can't find the container with id d5e6404860bc00d728f227b500e1b671a37db209b5bac79dea084ec7f07aef69 Sep 10 16:01:44.253105 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:44.253070 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4613e9_e2c0_4975_90e5_ce76c0e39814.slice/crio-61c688e19b232ef7bd1f49467b114bc2b16152701d293dc58d11c9e185d42699 WatchSource:0}: Error finding container 61c688e19b232ef7bd1f49467b114bc2b16152701d293dc58d11c9e185d42699: Status 404 returned error can't find the container with id 61c688e19b232ef7bd1f49467b114bc2b16152701d293dc58d11c9e185d42699 Sep 10 16:01:44.254766 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:01:44.254333 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54359979_f90b_4674_b39d_122f7c847055.slice/crio-9c490a0c323af2b4ec20b91ab30813da5866752d6d573f54969bc3d3890f7f87 WatchSource:0}: Error finding container 9c490a0c323af2b4ec20b91ab30813da5866752d6d573f54969bc3d3890f7f87: Status 404 returned error can't find the container with id 9c490a0c323af2b4ec20b91ab30813da5866752d6d573f54969bc3d3890f7f87 Sep 10 16:01:44.530606 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:44.530575 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqbq\" (UniqueName: \"kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq\") pod \"network-check-target-j2m6r\" (UID: \"ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e\") " pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:01:44.530754 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:44.530736 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 16:01:44.530808 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:44.530761 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 16:01:44.530808 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:44.530771 2541 projected.go:196] Error preparing data for projected volume kube-api-access-rhqbq for pod openshift-network-diagnostics/network-check-target-j2m6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:01:44.530889 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:44.530816 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq podName:ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e nodeName:}" failed. No retries permitted until 2026-09-10 16:01:46.53080096 +0000 UTC m=+6.054046137 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rhqbq" (UniqueName: "kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq") pod "network-check-target-j2m6r" (UID: "ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:01:44.631838 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:44.631633 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs\") pod \"network-metrics-daemon-h5frk\" (UID: \"086a5470-5fac-4be6-963c-7b219b25b740\") " pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:01:44.631977 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:44.631861 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret\") pod \"global-pull-secret-syncer-cjj7d\" (UID: \"e3074957-6ddd-4ded-9ddf-7c2fe68929f0\") " pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:01:44.631977 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:44.631784 2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:01:44.631977 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:44.631957 2541 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 16:01:44.631977 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:44.631975 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs podName:086a5470-5fac-4be6-963c-7b219b25b740 nodeName:}" failed. No retries permitted until 2026-09-10 16:01:46.631957226 +0000 UTC m=+6.155202402 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs") pod "network-metrics-daemon-h5frk" (UID: "086a5470-5fac-4be6-963c-7b219b25b740") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:01:44.632129 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:44.632004 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret podName:e3074957-6ddd-4ded-9ddf-7c2fe68929f0 nodeName:}" failed. No retries permitted until 2026-09-10 16:01:46.63199309 +0000 UTC m=+6.155238266 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret") pod "global-pull-secret-syncer-cjj7d" (UID: "e3074957-6ddd-4ded-9ddf-7c2fe68929f0") : object "kube-system"/"original-pull-secret" not registered Sep 10 16:01:44.954282 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:44.954234 2541 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 15:56:43 +0000 UTC" deadline="2028-05-31 02:14:18.985254066 +0000 UTC" Sep 10 16:01:44.954282 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:44.954274 2541 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15082h12m34.030985551s" Sep 10 16:01:44.977356 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:44.976551 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:01:44.977356 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:44.976710 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjj7d" podUID="e3074957-6ddd-4ded-9ddf-7c2fe68929f0" Sep 10 16:01:44.977356 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:44.977146 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:01:44.977356 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:44.977258 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2m6r" podUID="ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e" Sep 10 16:01:45.006741 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:45.006694 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9crl" event={"ID":"54359979-f90b-4674-b39d-122f7c847055","Type":"ContainerStarted","Data":"9c490a0c323af2b4ec20b91ab30813da5866752d6d573f54969bc3d3890f7f87"} Sep 10 16:01:45.011690 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:45.011653 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wjm74" event={"ID":"2e4613e9-e2c0-4975-90e5-ce76c0e39814","Type":"ContainerStarted","Data":"61c688e19b232ef7bd1f49467b114bc2b16152701d293dc58d11c9e185d42699"} Sep 10 16:01:45.014066 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:45.014033 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lqg46" event={"ID":"67781cee-0ff9-4fb7-9e88-056433782dd5","Type":"ContainerStarted","Data":"d5e6404860bc00d728f227b500e1b671a37db209b5bac79dea084ec7f07aef69"} Sep 10 16:01:45.015896 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:45.015846 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-48lj8" event={"ID":"40f27517-24fa-41df-8a4d-1eefd387cf5d","Type":"ContainerStarted","Data":"1467096ee3353dee2c3fcb1c84baa5207c3f48f867e63fa9326caa093db0410c"} Sep 10 16:01:45.018483 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:45.018456 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" event={"ID":"b95766bc-a06f-45f1-a0d0-63892b29d68f","Type":"ContainerStarted","Data":"e3e89177876a2c30698e4761245c90a058aec1275f997069c6ddf0b942becfc1"} Sep 10 16:01:45.023984 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:45.023952 2541 generic.go:356] "Generic (PLEG): container finished" podID="60e762d75cf3fcfb77ba57171376954a" containerID="a2e218311f1c862ca87039b0d9acebe69718a22343d99502c292013304aaa3a0" exitCode=0 Sep 10 16:01:45.024100 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:45.024032 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" event={"ID":"60e762d75cf3fcfb77ba57171376954a","Type":"ContainerDied","Data":"a2e218311f1c862ca87039b0d9acebe69718a22343d99502c292013304aaa3a0"} Sep 10 16:01:45.027344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:45.027320 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" event={"ID":"d67875b1-0368-48c9-92ac-daeb4f6f9b03","Type":"ContainerStarted","Data":"cb8052780f677b4d9cd7ac9c110f9b47d7abc0305949bbbb6892662135cb365f"} Sep 10 16:01:45.030886 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:45.030862 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" event={"ID":"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd","Type":"ContainerStarted","Data":"78c58a75d3457ed04a2d5ef76b3e36d911ec6cfc91ff78ff2e6f8ad4a4fc14d8"} Sep 10 16:01:45.037478 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:45.037397 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rxmz9" event={"ID":"7e5689cb-d3f5-4d8c-b91c-a0e40aeeeef9","Type":"ContainerStarted","Data":"1735a89c1eebdde95a8beeb6daf5c19fb4a9103c81be1cf52721fc3a1f3aad60"} Sep 10 16:01:45.040206 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:45.039746 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal" event={"ID":"93402cb6c1e59df178a12145f97bf120","Type":"ContainerStarted","Data":"a3d0f762e7335a4f45767c161dedff584059e07154a1d9d40047f5f3a80631a1"} Sep 10 16:01:45.974081 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:45.973563 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:01:45.974081 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:45.973710 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5frk" podUID="086a5470-5fac-4be6-963c-7b219b25b740" Sep 10 16:01:46.053608 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:46.053529 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" event={"ID":"60e762d75cf3fcfb77ba57171376954a","Type":"ContainerStarted","Data":"d236acf85fb3c38da98d4679040509ec1b48d215e1fbf5b8060ff4e309129bc1"} Sep 10 16:01:46.069798 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:46.069737 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" podStartSLOduration=4.069719891 podStartE2EDuration="4.069719891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:46.069173219 +0000 UTC m=+5.592418421" watchObservedRunningTime="2026-09-10 16:01:46.069719891 +0000 UTC m=+5.592965094" Sep 10 16:01:46.070651 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:46.070605 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal" podStartSLOduration=4.070590372 podStartE2EDuration="4.070590372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:45.075558133 +0000 UTC m=+4.598803335" watchObservedRunningTime="2026-09-10 16:01:46.070590372 +0000 UTC m=+5.593835575" Sep 10 16:01:46.549355 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:46.549300 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqbq\" (UniqueName: \"kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq\") pod \"network-check-target-j2m6r\" (UID: \"ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e\") " pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:01:46.549557 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:46.549537 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 16:01:46.549630 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:46.549563 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 16:01:46.549630 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:46.549578 2541 projected.go:196] Error preparing data for projected volume kube-api-access-rhqbq for pod openshift-network-diagnostics/network-check-target-j2m6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:01:46.549758 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:46.549667 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq podName:ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e nodeName:}" failed. No retries permitted until 2026-09-10 16:01:50.549633813 +0000 UTC m=+10.072878997 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rhqbq" (UniqueName: "kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq") pod "network-check-target-j2m6r" (UID: "ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:01:46.650419 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:46.650380 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs\") pod \"network-metrics-daemon-h5frk\" (UID: \"086a5470-5fac-4be6-963c-7b219b25b740\") " pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:01:46.650599 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:46.650441 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret\") pod \"global-pull-secret-syncer-cjj7d\" (UID: \"e3074957-6ddd-4ded-9ddf-7c2fe68929f0\") " pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:01:46.650685 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:46.650594 2541 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 16:01:46.650685 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:46.650661 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret podName:e3074957-6ddd-4ded-9ddf-7c2fe68929f0 nodeName:}" failed. No retries permitted until 2026-09-10 16:01:50.650641419 +0000 UTC m=+10.173886613 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret") pod "global-pull-secret-syncer-cjj7d" (UID: "e3074957-6ddd-4ded-9ddf-7c2fe68929f0") : object "kube-system"/"original-pull-secret" not registered Sep 10 16:01:46.651116 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:46.651095 2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:01:46.651199 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:46.651158 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs podName:086a5470-5fac-4be6-963c-7b219b25b740 nodeName:}" failed. No retries permitted until 2026-09-10 16:01:50.651138759 +0000 UTC m=+10.174383941 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs") pod "network-metrics-daemon-h5frk" (UID: "086a5470-5fac-4be6-963c-7b219b25b740") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:01:46.974594 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:46.974495 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:01:46.975061 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:46.974636 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2m6r" podUID="ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e" Sep 10 16:01:46.975061 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:46.974677 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:01:46.975061 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:46.974756 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjj7d" podUID="e3074957-6ddd-4ded-9ddf-7c2fe68929f0" Sep 10 16:01:47.973466 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:47.973419 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:01:47.973667 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:47.973575 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5frk" podUID="086a5470-5fac-4be6-963c-7b219b25b740" Sep 10 16:01:48.060568 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:48.060527 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-48lj8" event={"ID":"40f27517-24fa-41df-8a4d-1eefd387cf5d","Type":"ContainerStarted","Data":"4d45ede8f3881a46e8c51ec5c73490e2fa0b4a4a30d0caf45fa52b3ffde26910"} Sep 10 16:01:48.077173 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:48.077117 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-48lj8" podStartSLOduration=4.030263184 podStartE2EDuration="7.077094347s" totalImagesPullingTime="3.046831163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:48.076865431 +0000 UTC m=+7.600110632" watchObservedRunningTime="2026-09-10 16:01:48.077094347 +0000 UTC m=+7.600339548" Sep 10 16:01:48.976610 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:48.976054 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:01:48.976610 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:48.976194 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjj7d" podUID="e3074957-6ddd-4ded-9ddf-7c2fe68929f0" Sep 10 16:01:48.976610 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:48.976482 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:01:48.976610 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:48.976575 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2m6r" podUID="ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e" Sep 10 16:01:49.973362 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:49.973277 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:01:49.973953 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:49.973458 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5frk" podUID="086a5470-5fac-4be6-963c-7b219b25b740" Sep 10 16:01:50.583995 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:50.583951 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqbq\" (UniqueName: \"kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq\") pod \"network-check-target-j2m6r\" (UID: \"ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e\") " pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:01:50.584199 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:50.584153 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 16:01:50.584199 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:50.584174 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 16:01:50.584199 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:50.584191 2541 projected.go:196] Error preparing data for projected volume kube-api-access-rhqbq for pod openshift-network-diagnostics/network-check-target-j2m6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:01:50.584354 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:50.584255 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq podName:ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e nodeName:}" failed. No retries permitted until 2026-09-10 16:01:58.584235221 +0000 UTC m=+18.107480404 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rhqbq" (UniqueName: "kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq") pod "network-check-target-j2m6r" (UID: "ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:01:50.685080 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:50.685037 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs\") pod \"network-metrics-daemon-h5frk\" (UID: \"086a5470-5fac-4be6-963c-7b219b25b740\") " pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:01:50.685283 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:50.685101 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret\") pod \"global-pull-secret-syncer-cjj7d\" (UID: \"e3074957-6ddd-4ded-9ddf-7c2fe68929f0\") " pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:01:50.685283 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:50.685272 2541 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 16:01:50.685428 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:50.685326 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret podName:e3074957-6ddd-4ded-9ddf-7c2fe68929f0 nodeName:}" failed. No retries permitted until 2026-09-10 16:01:58.685311455 +0000 UTC m=+18.208556646 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret") pod "global-pull-secret-syncer-cjj7d" (UID: "e3074957-6ddd-4ded-9ddf-7c2fe68929f0") : object "kube-system"/"original-pull-secret" not registered Sep 10 16:01:50.685428 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:50.685408 2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:01:50.685552 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:50.685448 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs podName:086a5470-5fac-4be6-963c-7b219b25b740 nodeName:}" failed. No retries permitted until 2026-09-10 16:01:58.685434585 +0000 UTC m=+18.208679770 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs") pod "network-metrics-daemon-h5frk" (UID: "086a5470-5fac-4be6-963c-7b219b25b740") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:01:50.976340 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:50.975687 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:01:50.976340 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:50.975855 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjj7d" podUID="e3074957-6ddd-4ded-9ddf-7c2fe68929f0" Sep 10 16:01:50.976340 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:50.975922 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:01:50.976340 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:50.976064 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2m6r" podUID="ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e" Sep 10 16:01:51.973646 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:51.973613 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:01:51.973848 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:51.973751 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5frk" podUID="086a5470-5fac-4be6-963c-7b219b25b740" Sep 10 16:01:52.624586 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:52.624551 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-48lj8" Sep 10 16:01:52.625339 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:52.625087 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-48lj8" Sep 10 16:01:52.974501 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:52.974418 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:01:52.974661 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:52.974533 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjj7d" podUID="e3074957-6ddd-4ded-9ddf-7c2fe68929f0" Sep 10 16:01:52.975085 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:52.974928 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:01:52.975085 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:52.975015 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2m6r" podUID="ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e" Sep 10 16:01:53.973921 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:53.973882 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:01:53.974414 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:53.974029 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5frk" podUID="086a5470-5fac-4be6-963c-7b219b25b740" Sep 10 16:01:54.973255 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:54.973216 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:01:54.973466 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:54.973257 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:01:54.973466 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:54.973369 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2m6r" podUID="ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e" Sep 10 16:01:54.973577 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:54.973495 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjj7d" podUID="e3074957-6ddd-4ded-9ddf-7c2fe68929f0" Sep 10 16:01:55.973611 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:55.973575 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:01:55.974031 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:55.973708 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5frk" podUID="086a5470-5fac-4be6-963c-7b219b25b740" Sep 10 16:01:56.972954 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:56.972896 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:01:56.973158 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:56.973024 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjj7d" podUID="e3074957-6ddd-4ded-9ddf-7c2fe68929f0" Sep 10 16:01:56.973158 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:56.973082 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:01:56.973285 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:56.973176 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2m6r" podUID="ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e" Sep 10 16:01:57.973139 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:57.973100 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:01:57.973582 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:57.973250 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5frk" podUID="086a5470-5fac-4be6-963c-7b219b25b740" Sep 10 16:01:58.646873 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:58.646834 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqbq\" (UniqueName: \"kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq\") pod \"network-check-target-j2m6r\" (UID: \"ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e\") " pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:01:58.647064 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:58.647026 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 16:01:58.647064 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:58.647048 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 16:01:58.647064 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:58.647058 2541 projected.go:196] Error preparing data for projected volume kube-api-access-rhqbq for pod openshift-network-diagnostics/network-check-target-j2m6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:01:58.647208 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:58.647117 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq podName:ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e nodeName:}" failed. No retries permitted until 2026-09-10 16:02:14.647099492 +0000 UTC m=+34.170344690 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rhqbq" (UniqueName: "kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq") pod "network-check-target-j2m6r" (UID: "ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:01:58.747804 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:58.747761 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret\") pod \"global-pull-secret-syncer-cjj7d\" (UID: \"e3074957-6ddd-4ded-9ddf-7c2fe68929f0\") " pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:01:58.747974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:58.747862 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs\") pod \"network-metrics-daemon-h5frk\" (UID: \"086a5470-5fac-4be6-963c-7b219b25b740\") " pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:01:58.747974 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:58.747941 2541 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 16:01:58.747974 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:58.747967 2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:01:58.748083 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:58.748014 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret podName:e3074957-6ddd-4ded-9ddf-7c2fe68929f0 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:14.747995813 +0000 UTC m=+34.271241008 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret") pod "global-pull-secret-syncer-cjj7d" (UID: "e3074957-6ddd-4ded-9ddf-7c2fe68929f0") : object "kube-system"/"original-pull-secret" not registered Sep 10 16:01:58.748083 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:58.748030 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs podName:086a5470-5fac-4be6-963c-7b219b25b740 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:14.748023062 +0000 UTC m=+34.271268237 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs") pod "network-metrics-daemon-h5frk" (UID: "086a5470-5fac-4be6-963c-7b219b25b740") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:01:58.973207 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:58.973128 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:01:58.973630 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:58.973247 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjj7d" podUID="e3074957-6ddd-4ded-9ddf-7c2fe68929f0" Sep 10 16:01:58.973630 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:58.973330 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:01:58.973630 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:58.973439 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2m6r" podUID="ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e" Sep 10 16:01:59.973889 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:01:59.973851 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:01:59.974320 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:01:59.974007 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5frk" podUID="086a5470-5fac-4be6-963c-7b219b25b740" Sep 10 16:02:00.974120 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:00.974091 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:02:00.974539 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:00.974173 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjj7d" podUID="e3074957-6ddd-4ded-9ddf-7c2fe68929f0" Sep 10 16:02:00.974539 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:00.974179 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:02:00.974539 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:00.974282 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2m6r" podUID="ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e" Sep 10 16:02:01.973206 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:01.973157 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:02:01.973434 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:01.973322 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5frk" podUID="086a5470-5fac-4be6-963c-7b219b25b740" Sep 10 16:02:02.976938 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:02.976753 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:02:02.977357 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:02.976789 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:02:02.977357 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:02.977017 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2m6r" podUID="ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e" Sep 10 16:02:02.977357 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:02.977128 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjj7d" podUID="e3074957-6ddd-4ded-9ddf-7c2fe68929f0" Sep 10 16:02:03.090670 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:03.090637 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" event={"ID":"d67875b1-0368-48c9-92ac-daeb4f6f9b03","Type":"ContainerStarted","Data":"74e98056f925589b105edfcda1cd23118795724888696148a28eef2190bb4c6c"} Sep 10 16:02:03.092094 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:03.092067 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rxmz9" event={"ID":"7e5689cb-d3f5-4d8c-b91c-a0e40aeeeef9","Type":"ContainerStarted","Data":"fcc8d2e7739eca386f5967a92eca6c215a046767d965c80c688f71a53161e10c"} Sep 10 16:02:03.093588 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:03.093554 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9crl" event={"ID":"54359979-f90b-4674-b39d-122f7c847055","Type":"ContainerStarted","Data":"3e0672a10224287fde668a6fdd4b295afd3660ae044ef38aabfcf8932146d5f7"} Sep 10 16:02:03.095070 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:03.095036 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wjm74" event={"ID":"2e4613e9-e2c0-4975-90e5-ce76c0e39814","Type":"ContainerStarted","Data":"51cbfee960f7093acd84dd2b8f27e934350fb1a857d186bea27b2ddf7a33f1d7"} Sep 10 16:02:03.099127 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:03.099102 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" event={"ID":"b95766bc-a06f-45f1-a0d0-63892b29d68f","Type":"ContainerStarted","Data":"c3f399d80192070e9027a3cc39151461b44858077d12e6a9d45c408f01d8bcde"} Sep 10 16:02:03.105089 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:03.105038 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rxmz9" podStartSLOduration=2.819882319 podStartE2EDuration="21.105025281s" totalImagesPullingTime="18.285142962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:03.104876265 +0000 UTC m=+22.628121464" watchObservedRunningTime="2026-09-10 16:02:03.105025281 +0000 UTC m=+22.628270477" Sep 10 16:02:03.121575 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:03.121515 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wjm74" podStartSLOduration=3.777594315 podStartE2EDuration="22.121496137s" totalImagesPullingTime="18.343901822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:03.121323149 +0000 UTC m=+22.644568342" watchObservedRunningTime="2026-09-10 16:02:03.121496137 +0000 UTC m=+22.644741336" Sep 10 16:02:03.140168 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:03.140122 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7gfgn" podStartSLOduration=2.827233562 podStartE2EDuration="21.140105445s" totalImagesPullingTime="18.312871883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:03.139611825 +0000 UTC m=+22.662857023" watchObservedRunningTime="2026-09-10 16:02:03.140105445 +0000 UTC m=+22.663350641" Sep 10 16:02:03.973310 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:03.973108 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:02:03.973489 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:03.973404 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5frk" podUID="086a5470-5fac-4be6-963c-7b219b25b740" Sep 10 16:02:04.067726 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:04.067699 2541 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 16:02:04.102848 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:04.102817 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lqg46" event={"ID":"67781cee-0ff9-4fb7-9e88-056433782dd5","Type":"ContainerStarted","Data":"1e60c1fc5e894de839fc84abddfabea5b3773f70f5bb2309b0759823a42d49b3"} Sep 10 16:02:04.104772 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:04.104741 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" event={"ID":"d67875b1-0368-48c9-92ac-daeb4f6f9b03","Type":"ContainerStarted","Data":"b5c3ea0e7959ea7a05b317e5f30d3ddfbaaa12ef1c5f1d7c4e8c5fbce9c539c9"} Sep 10 16:02:04.107607 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:04.107577 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4c6j_2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd/ovn-acl-logging/0.log" Sep 10 16:02:04.107911 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:04.107891 2541 generic.go:356] "Generic (PLEG): container finished" podID="2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd" containerID="21965236cf1b5f71d0f9a2d4be797a601ccc9680e5f58932c46e1da705a71a6c" exitCode=1 Sep 10 16:02:04.107973 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:04.107960 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" event={"ID":"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd","Type":"ContainerStarted","Data":"1df5d86ebbb42231ba31c860cc6ff56c89b5597523f888b3fb818fe2074dd2d4"} Sep 10 16:02:04.108037 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:04.108024 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" event={"ID":"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd","Type":"ContainerStarted","Data":"b07e4d4f35bf167e272c18a0e6ae4d5277c6f57a2d07f7db1718aec8ba7713c6"} Sep 10 16:02:04.108091 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:04.108079 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" event={"ID":"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd","Type":"ContainerStarted","Data":"cd88a48dd3439885a791ded036a877f941a2695ee5dcf75f0189f335f2bbf875"} Sep 10 16:02:04.108129 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:04.108119 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" event={"ID":"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd","Type":"ContainerStarted","Data":"3bb28027e32a8df9007603cb2c5b924c1218aac5f30c9e7af7aa912a752d6242"} Sep 10 16:02:04.108180 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:04.108163 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" event={"ID":"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd","Type":"ContainerDied","Data":"21965236cf1b5f71d0f9a2d4be797a601ccc9680e5f58932c46e1da705a71a6c"} Sep 10 16:02:04.108232 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:04.108197 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" event={"ID":"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd","Type":"ContainerStarted","Data":"0ef312c6cf144d1bc2c125a6b9cff1d21fecd1e6d414be2fd614c0935a894908"} Sep 10 16:02:04.109271 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:04.109251 2541 generic.go:356] "Generic (PLEG): container finished" podID="54359979-f90b-4674-b39d-122f7c847055" containerID="3e0672a10224287fde668a6fdd4b295afd3660ae044ef38aabfcf8932146d5f7" exitCode=0 Sep 10 16:02:04.109385 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:04.109350 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9crl" event={"ID":"54359979-f90b-4674-b39d-122f7c847055","Type":"ContainerDied","Data":"3e0672a10224287fde668a6fdd4b295afd3660ae044ef38aabfcf8932146d5f7"} Sep 10 16:02:04.134737 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:04.134686 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lqg46" podStartSLOduration=4.830495853 podStartE2EDuration="23.134669423s" totalImagesPullingTime="18.30417357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:04.134285641 +0000 UTC m=+23.657530837" watchObservedRunningTime="2026-09-10 16:02:04.134669423 +0000 UTC m=+23.657914619" Sep 10 16:02:04.973171 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:04.973138 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:02:04.973171 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:04.973160 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:02:04.973432 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:04.973281 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjj7d" podUID="e3074957-6ddd-4ded-9ddf-7c2fe68929f0" Sep 10 16:02:04.973432 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:04.973402 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2m6r" podUID="ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e" Sep 10 16:02:04.976283 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:04.976189 2541 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T16:02:04.067719472Z","UUID":"e67719bb-0ebc-49a1-904d-aa9eec2ec324","Handler":null,"Name":"","Endpoint":""} Sep 10 16:02:04.977948 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:04.977930 2541 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 16:02:04.978054 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:04.977957 2541 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 16:02:05.973007 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:05.972976 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:02:05.973447 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:05.973092 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5frk" podUID="086a5470-5fac-4be6-963c-7b219b25b740" Sep 10 16:02:06.116695 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:06.116604 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" event={"ID":"d67875b1-0368-48c9-92ac-daeb4f6f9b03","Type":"ContainerStarted","Data":"aeafc57319ca76b3d3f97d4378a39e658cf67db0a638e2897036df74de10cc3a"} Sep 10 16:02:06.120282 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:06.120258 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4c6j_2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd/ovn-acl-logging/0.log" Sep 10 16:02:06.120672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:06.120649 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" event={"ID":"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd","Type":"ContainerStarted","Data":"b7578f87eaa119c222109df941a5d73207e954c50572962ec339aca7e1a1822f"} Sep 10 16:02:06.140288 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:06.140208 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ct9xc" podStartSLOduration=3.569588402 podStartE2EDuration="24.140186839s" totalImagesPullingTime="20.570598437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:01:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:06.139756434 +0000 UTC m=+25.663001656" watchObservedRunningTime="2026-09-10 16:02:06.140186839 +0000 UTC m=+25.663432064" Sep 10 16:02:06.785914 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:06.785879 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-48lj8" Sep 10 16:02:06.786068 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:06.786040 2541 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 10 16:02:06.786503 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:06.786478 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-48lj8" Sep 10 16:02:06.973316 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:06.973277 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:02:06.973784 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:06.973277 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:02:06.973784 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:06.973420 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2m6r" podUID="ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e" Sep 10 16:02:06.973784 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:06.973503 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjj7d" podUID="e3074957-6ddd-4ded-9ddf-7c2fe68929f0" Sep 10 16:02:07.973676 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:07.973641 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:02:07.974056 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:07.973762 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5frk" podUID="086a5470-5fac-4be6-963c-7b219b25b740" Sep 10 16:02:08.973713 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:08.973680 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:02:08.974381 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:08.973786 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjj7d" podUID="e3074957-6ddd-4ded-9ddf-7c2fe68929f0" Sep 10 16:02:08.974381 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:08.973846 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:02:08.974381 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:08.973931 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2m6r" podUID="ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e" Sep 10 16:02:09.129071 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:09.129041 2541 generic.go:356] "Generic (PLEG): container finished" podID="54359979-f90b-4674-b39d-122f7c847055" containerID="38ac2043c711b8cb6babbf45029a4e5448c4c6c3cf900ea2f685c652b702a3f0" exitCode=0 Sep 10 16:02:09.129225 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:09.129145 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9crl" event={"ID":"54359979-f90b-4674-b39d-122f7c847055","Type":"ContainerDied","Data":"38ac2043c711b8cb6babbf45029a4e5448c4c6c3cf900ea2f685c652b702a3f0"} Sep 10 16:02:09.132728 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:09.132711 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4c6j_2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd/ovn-acl-logging/0.log" Sep 10 16:02:09.133051 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:09.133033 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" event={"ID":"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd","Type":"ContainerStarted","Data":"e7b08dde6e62992d0d0f7e172b586e909c10396e69ff96dd2b9d5b414bb6a052"} Sep 10 16:02:09.133405 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:09.133384 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:02:09.133508 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:09.133440 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:02:09.133573 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:09.133532 2541 scope.go:117] "RemoveContainer" containerID="21965236cf1b5f71d0f9a2d4be797a601ccc9680e5f58932c46e1da705a71a6c" Sep 10 16:02:09.149942 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:09.149922 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:02:09.972979 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:09.972951 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:02:09.973110 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:09.973054 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5frk" podUID="086a5470-5fac-4be6-963c-7b219b25b740" Sep 10 16:02:10.140106 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:10.140071 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4c6j_2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd/ovn-acl-logging/0.log" Sep 10 16:02:10.140715 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:10.140482 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" event={"ID":"2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd","Type":"ContainerStarted","Data":"d052df764efafe71482a07f17e0a7c12328d7b13402167fa99f07955c17b8ae2"} Sep 10 16:02:10.140910 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:10.140892 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:02:10.158917 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:10.158885 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:02:10.170016 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:10.169463 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" podStartSLOduration=9.502275987 podStartE2EDuration="28.169441361s" totalImagesPullingTime="18.667165374s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:10.168857005 +0000 UTC m=+29.692102210" watchObservedRunningTime="2026-09-10 16:02:10.169441361 +0000 UTC m=+29.692686558" Sep 10 16:02:10.250453 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:10.250375 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cjj7d"] Sep 10 16:02:10.250598 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:10.250544 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:02:10.250683 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:10.250655 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjj7d" podUID="e3074957-6ddd-4ded-9ddf-7c2fe68929f0" Sep 10 16:02:10.253446 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:10.253413 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h5frk"] Sep 10 16:02:10.253560 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:10.253522 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:02:10.253646 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:10.253620 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5frk" podUID="086a5470-5fac-4be6-963c-7b219b25b740" Sep 10 16:02:10.254092 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:10.254072 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j2m6r"] Sep 10 16:02:10.254184 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:10.254173 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:02:10.254280 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:10.254261 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2m6r" podUID="ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e" Sep 10 16:02:11.145099 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:11.144892 2541 generic.go:356] "Generic (PLEG): container finished" podID="54359979-f90b-4674-b39d-122f7c847055" containerID="2921401a1937b955f419d216445fe7def3fd843f532c55caaef966c895f8dbfb" exitCode=0 Sep 10 16:02:11.145462 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:11.144967 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9crl" event={"ID":"54359979-f90b-4674-b39d-122f7c847055","Type":"ContainerDied","Data":"2921401a1937b955f419d216445fe7def3fd843f532c55caaef966c895f8dbfb"} Sep 10 16:02:11.973835 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:11.973810 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:02:11.973983 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:11.973811 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:02:11.974019 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:11.973815 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:02:11.974019 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:11.973990 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5frk" podUID="086a5470-5fac-4be6-963c-7b219b25b740" Sep 10 16:02:11.974087 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:11.973895 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2m6r" podUID="ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e" Sep 10 16:02:11.974121 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:11.974094 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjj7d" podUID="e3074957-6ddd-4ded-9ddf-7c2fe68929f0" Sep 10 16:02:12.152266 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:12.152233 2541 generic.go:356] "Generic (PLEG): container finished" podID="54359979-f90b-4674-b39d-122f7c847055" containerID="47c9ff424fa5a3e2d1cb2f612865edf90965b99353378f586cee4d30f8e36f65" exitCode=0 Sep 10 16:02:12.152647 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:12.152323 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9crl" event={"ID":"54359979-f90b-4674-b39d-122f7c847055","Type":"ContainerDied","Data":"47c9ff424fa5a3e2d1cb2f612865edf90965b99353378f586cee4d30f8e36f65"} Sep 10 16:02:13.973350 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:13.973317 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:02:13.973350 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:13.973350 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:02:13.973822 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:13.973320 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:02:13.973822 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:13.973436 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2m6r" podUID="ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e" Sep 10 16:02:13.973822 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:13.973572 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjj7d" podUID="e3074957-6ddd-4ded-9ddf-7c2fe68929f0" Sep 10 16:02:13.973822 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:13.973653 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5frk" podUID="086a5470-5fac-4be6-963c-7b219b25b740" Sep 10 16:02:14.662477 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:14.662442 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqbq\" (UniqueName: \"kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq\") pod \"network-check-target-j2m6r\" (UID: \"ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e\") " pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:02:14.662633 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:14.662597 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 16:02:14.662633 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:14.662619 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 16:02:14.662633 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:14.662632 2541 projected.go:196] Error preparing data for projected volume kube-api-access-rhqbq for pod openshift-network-diagnostics/network-check-target-j2m6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:02:14.662729 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:14.662690 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq podName:ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e nodeName:}" failed. No retries permitted until 2026-09-10 16:02:46.662672029 +0000 UTC m=+66.185917221 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rhqbq" (UniqueName: "kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq") pod "network-check-target-j2m6r" (UID: "ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:02:14.763212 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:14.763173 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs\") pod \"network-metrics-daemon-h5frk\" (UID: \"086a5470-5fac-4be6-963c-7b219b25b740\") " pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:02:14.763424 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:14.763246 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret\") pod \"global-pull-secret-syncer-cjj7d\" (UID: \"e3074957-6ddd-4ded-9ddf-7c2fe68929f0\") " pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:02:14.763424 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:14.763351 2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:02:14.763424 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:14.763394 2541 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 16:02:14.763424 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:14.763425 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs podName:086a5470-5fac-4be6-963c-7b219b25b740 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:46.763404264 +0000 UTC m=+66.286649439 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs") pod "network-metrics-daemon-h5frk" (UID: "086a5470-5fac-4be6-963c-7b219b25b740") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:02:14.763602 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:14.763451 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret podName:e3074957-6ddd-4ded-9ddf-7c2fe68929f0 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:46.763435966 +0000 UTC m=+66.286681164 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret") pod "global-pull-secret-syncer-cjj7d" (UID: "e3074957-6ddd-4ded-9ddf-7c2fe68929f0") : object "kube-system"/"original-pull-secret" not registered Sep 10 16:02:15.973231 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:15.973197 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:02:15.973808 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:15.973210 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:02:15.973808 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:15.973325 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5frk" podUID="086a5470-5fac-4be6-963c-7b219b25b740" Sep 10 16:02:15.973808 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:15.973341 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:02:15.973808 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:15.973418 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2m6r" podUID="ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e" Sep 10 16:02:15.973808 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:15.973495 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjj7d" podUID="e3074957-6ddd-4ded-9ddf-7c2fe68929f0" Sep 10 16:02:16.595976 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.595895 2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeReady" Sep 10 16:02:16.596111 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.596029 2541 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 16:02:16.669108 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.669080 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65b9595f8c-v7mjc"] Sep 10 16:02:16.697040 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.697005 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-scdqf"] Sep 10 16:02:16.697916 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.697896 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.705704 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.705678 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nqhnk\"" Sep 10 16:02:16.706031 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.706013 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 10 16:02:16.706627 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.706609 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 10 16:02:16.707160 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.707141 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 10 16:02:16.712990 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.712955 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-bt6g6"] Sep 10 16:02:16.713120 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.713098 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-scdqf" Sep 10 16:02:16.730652 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.730613 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 10 16:02:16.730807 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.730689 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 10 16:02:16.730961 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.730947 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-866xb\"" Sep 10 16:02:16.731888 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.731870 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 10 16:02:16.734023 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.734004 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm"] Sep 10 16:02:16.734163 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.734139 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-bt6g6" Sep 10 16:02:16.736748 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.736728 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 10 16:02:16.736844 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.736826 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 10 16:02:16.736985 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.736968 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7j2sr\"" Sep 10 16:02:16.737984 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.737954 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 10 16:02:16.738789 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.738762 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 10 16:02:16.746169 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.746142 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 10 16:02:16.750812 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.750780 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sl7bk"] Sep 10 16:02:16.750922 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.750894 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm" Sep 10 16:02:16.755788 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.755766 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 16:02:16.756236 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.756219 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-llg5z\"" Sep 10 16:02:16.756534 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.756505 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 10 16:02:16.756673 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.756657 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 16:02:16.757314 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.757285 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 10 16:02:16.769111 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.769089 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7f48d5849b-h27lv"] Sep 10 16:02:16.769234 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.769214 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sl7bk" Sep 10 16:02:16.771324 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.771285 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 10 16:02:16.771600 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.771559 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 10 16:02:16.772519 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.772503 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gm4g7\"" Sep 10 16:02:16.773215 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.773195 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 10 16:02:16.776863 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.776832 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-bound-sa-token\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.776940 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.776871 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-image-registry-private-configuration\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.777023 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.776908 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-certificates\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.777077 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.777062 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.777112 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.777089 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-trusted-ca\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.777144 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.777122 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64mq5\" (UniqueName: \"kubernetes.io/projected/ae665a3c-336f-4057-8e8e-ca9572f1fd58-kube-api-access-64mq5\") pod \"volume-data-source-validator-695c895bd6-scdqf\" (UID: \"ae665a3c-336f-4057-8e8e-ca9572f1fd58\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-scdqf" Sep 10 16:02:16.777177 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.777149 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-installation-pull-secrets\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.777208 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.777180 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-ca-trust-extracted\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.777242 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.777226 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbp48\" (UniqueName: \"kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-kube-api-access-dbp48\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.785059 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.785035 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-89xqs"] Sep 10 16:02:16.785168 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.785147 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:16.787212 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.787150 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 10 16:02:16.787212 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.787158 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 10 16:02:16.787438 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.787285 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 10 16:02:16.787438 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.787433 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-l5xqb\"" Sep 10 16:02:16.787754 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.787738 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 16:02:16.787754 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.787744 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 16:02:16.787893 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.787775 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 10 16:02:16.811688 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.811649 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65b9595f8c-v7mjc"] Sep 10 16:02:16.811688 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.811687 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sl7bk"] Sep 10 16:02:16.811916 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.811713 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-scdqf"] Sep 10 16:02:16.811916 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.811734 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f48d5849b-h27lv"] Sep 10 16:02:16.811916 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.811753 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-89xqs"] Sep 10 16:02:16.811916 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.811773 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pk2xn"] Sep 10 16:02:16.811916 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.811783 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" Sep 10 16:02:16.814132 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.814107 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 16:02:16.814132 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.814121 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 10 16:02:16.814355 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.814186 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8zpsp\"" Sep 10 16:02:16.814355 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.814107 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 16:02:16.814355 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.814116 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 10 16:02:16.821918 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.821888 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 10 16:02:16.827734 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.827705 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-bt6g6"] Sep 10 16:02:16.827869 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.827780 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm"] Sep 10 16:02:16.827869 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.827838 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pk2xn"] Sep 10 16:02:16.827869 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.827853 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pk2xn" Sep 10 16:02:16.827869 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.827868 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6r7v6"] Sep 10 16:02:16.829801 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.829776 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 16:02:16.829936 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.829827 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 16:02:16.829936 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.829834 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-twhfc\"" Sep 10 16:02:16.829936 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.829827 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 16:02:16.848067 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.847993 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6r7v6"] Sep 10 16:02:16.848180 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.848113 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6r7v6" Sep 10 16:02:16.850188 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.850164 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fqpr5\"" Sep 10 16:02:16.850336 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.850213 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 16:02:16.850336 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.850281 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 16:02:16.850336 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.850291 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 16:02:16.850515 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.850169 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 16:02:16.877777 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.877750 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64mq5\" (UniqueName: \"kubernetes.io/projected/ae665a3c-336f-4057-8e8e-ca9572f1fd58-kube-api-access-64mq5\") pod \"volume-data-source-validator-695c895bd6-scdqf\" (UID: \"ae665a3c-336f-4057-8e8e-ca9572f1fd58\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-scdqf" Sep 10 16:02:16.877907 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.877789 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b-config\") pod \"console-operator-85b454874b-bt6g6\" (UID: \"70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b\") " pod="openshift-console-operator/console-operator-85b454874b-bt6g6" Sep 10 16:02:16.877907 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.877807 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/befb7be9-0852-4486-a536-219bb03d51f6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-4nbnm\" (UID: \"befb7be9-0852-4486-a536-219bb03d51f6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm" Sep 10 16:02:16.877907 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.877829 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-installation-pull-secrets\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.877907 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.877854 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpdtv\" (UniqueName: \"kubernetes.io/projected/70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b-kube-api-access-kpdtv\") pod \"console-operator-85b454874b-bt6g6\" (UID: \"70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b\") " pod="openshift-console-operator/console-operator-85b454874b-bt6g6" Sep 10 16:02:16.877907 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.877907 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-ca-trust-extracted\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.878130 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.877933 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a5e2ec6-0e23-4d13-a153-25d193930e01-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-89xqs\" (UID: \"3a5e2ec6-0e23-4d13-a153-25d193930e01\") " pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" Sep 10 16:02:16.878130 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.877988 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbp48\" (UniqueName: \"kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-kube-api-access-dbp48\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.878130 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.878011 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-default-certificate\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:16.878130 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.878030 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39f061d0-48b5-40f0-8428-34d2a6bcc612-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sl7bk\" (UID: \"39f061d0-48b5-40f0-8428-34d2a6bcc612\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sl7bk" Sep 10 16:02:16.878130 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.878051 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txgp4\" (UniqueName: \"kubernetes.io/projected/39f061d0-48b5-40f0-8428-34d2a6bcc612-kube-api-access-txgp4\") pod \"cluster-samples-operator-7cdcfcf76c-sl7bk\" (UID: \"39f061d0-48b5-40f0-8428-34d2a6bcc612\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sl7bk" Sep 10 16:02:16.878130 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.878075 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a5e2ec6-0e23-4d13-a153-25d193930e01-tmp\") pod \"insights-operator-76bcd6ccf5-89xqs\" (UID: \"3a5e2ec6-0e23-4d13-a153-25d193930e01\") " pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" Sep 10 16:02:16.878440 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.878141 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78552\" (UniqueName: \"kubernetes.io/projected/3a5e2ec6-0e23-4d13-a153-25d193930e01-kube-api-access-78552\") pod \"insights-operator-76bcd6ccf5-89xqs\" (UID: \"3a5e2ec6-0e23-4d13-a153-25d193930e01\") " pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" Sep 10 16:02:16.878440 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.878222 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b-serving-cert\") pod \"console-operator-85b454874b-bt6g6\" (UID: \"70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b\") " pod="openshift-console-operator/console-operator-85b454874b-bt6g6" Sep 10 16:02:16.878440 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.878293 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-metrics-certs\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:16.878440 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.878330 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-ca-trust-extracted\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.878440 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.878375 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-bound-sa-token\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.878440 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.878403 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a5e2ec6-0e23-4d13-a153-25d193930e01-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-89xqs\" (UID: \"3a5e2ec6-0e23-4d13-a153-25d193930e01\") " pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" Sep 10 16:02:16.878440 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.878428 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a5e2ec6-0e23-4d13-a153-25d193930e01-serving-cert\") pod \"insights-operator-76bcd6ccf5-89xqs\" (UID: \"3a5e2ec6-0e23-4d13-a153-25d193930e01\") " pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" Sep 10 16:02:16.878776 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.878456 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-image-registry-private-configuration\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.878776 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.878482 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdd6746a-7957-4de4-8241-4c1d740f7814-service-ca-bundle\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:16.878776 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.878507 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-stats-auth\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:16.878776 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.878532 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6ldl\" (UniqueName: \"kubernetes.io/projected/fdd6746a-7957-4de4-8241-4c1d740f7814-kube-api-access-b6ldl\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:16.878776 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.878557 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8npjc\" (UniqueName: \"kubernetes.io/projected/befb7be9-0852-4486-a536-219bb03d51f6-kube-api-access-8npjc\") pod \"cluster-monitoring-operator-6cdcc96965-4nbnm\" (UID: \"befb7be9-0852-4486-a536-219bb03d51f6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm" Sep 10 16:02:16.878776 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.878587 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b-trusted-ca\") pod \"console-operator-85b454874b-bt6g6\" (UID: \"70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b\") " pod="openshift-console-operator/console-operator-85b454874b-bt6g6" Sep 10 16:02:16.878776 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.878618 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/befb7be9-0852-4486-a536-219bb03d51f6-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-4nbnm\" (UID: \"befb7be9-0852-4486-a536-219bb03d51f6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm" Sep 10 16:02:16.878776 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.878677 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-certificates\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.878776 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.878718 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.878776 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.878742 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-trusted-ca\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.878776 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.878769 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3a5e2ec6-0e23-4d13-a153-25d193930e01-snapshots\") pod \"insights-operator-76bcd6ccf5-89xqs\" (UID: \"3a5e2ec6-0e23-4d13-a153-25d193930e01\") " pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" Sep 10 16:02:16.879292 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:16.878867 2541 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 16:02:16.879292 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:16.878885 2541 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65b9595f8c-v7mjc: secret "image-registry-tls" not found Sep 10 16:02:16.879292 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:16.878951 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls podName:13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad nodeName:}" failed. No retries permitted until 2026-09-10 16:02:17.378933483 +0000 UTC m=+36.902178676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls") pod "image-registry-65b9595f8c-v7mjc" (UID: "13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad") : secret "image-registry-tls" not found Sep 10 16:02:16.879467 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.879341 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-certificates\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.879840 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.879817 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-trusted-ca\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.882337 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.882312 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-image-registry-private-configuration\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.882431 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.882336 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-installation-pull-secrets\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.893587 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.893551 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:16.895659 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.895623 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-bound-sa-token\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.895883 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.895862 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64mq5\" (UniqueName: \"kubernetes.io/projected/ae665a3c-336f-4057-8e8e-ca9572f1fd58-kube-api-access-64mq5\") pod \"volume-data-source-validator-695c895bd6-scdqf\" (UID: \"ae665a3c-336f-4057-8e8e-ca9572f1fd58\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-scdqf" Sep 10 16:02:16.896060 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.896039 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbp48\" (UniqueName: \"kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-kube-api-access-dbp48\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:16.980018 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.979984 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a5e2ec6-0e23-4d13-a153-25d193930e01-serving-cert\") pod \"insights-operator-76bcd6ccf5-89xqs\" (UID: \"3a5e2ec6-0e23-4d13-a153-25d193930e01\") " pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" Sep 10 16:02:16.980018 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.980017 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-tmp-dir\") pod \"dns-default-6r7v6\" (UID: \"0c3d3154-d3b2-4886-9bb6-0036bc9bdd35\") " pod="openshift-dns/dns-default-6r7v6" Sep 10 16:02:16.980524 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.980033 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdd6746a-7957-4de4-8241-4c1d740f7814-service-ca-bundle\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:16.980524 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.980050 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6ldl\" (UniqueName: \"kubernetes.io/projected/fdd6746a-7957-4de4-8241-4c1d740f7814-kube-api-access-b6ldl\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:16.980524 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.980266 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8npjc\" (UniqueName: \"kubernetes.io/projected/befb7be9-0852-4486-a536-219bb03d51f6-kube-api-access-8npjc\") pod \"cluster-monitoring-operator-6cdcc96965-4nbnm\" (UID: \"befb7be9-0852-4486-a536-219bb03d51f6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm" Sep 10 16:02:16.980524 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.980284 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-config-volume\") pod \"dns-default-6r7v6\" (UID: \"0c3d3154-d3b2-4886-9bb6-0036bc9bdd35\") " pod="openshift-dns/dns-default-6r7v6" Sep 10 16:02:16.980524 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.980333 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3a5e2ec6-0e23-4d13-a153-25d193930e01-snapshots\") pod \"insights-operator-76bcd6ccf5-89xqs\" (UID: \"3a5e2ec6-0e23-4d13-a153-25d193930e01\") " pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" Sep 10 16:02:16.980524 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:16.980396 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdd6746a-7957-4de4-8241-4c1d740f7814-service-ca-bundle podName:fdd6746a-7957-4de4-8241-4c1d740f7814 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:17.480373616 +0000 UTC m=+37.003618808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fdd6746a-7957-4de4-8241-4c1d740f7814-service-ca-bundle") pod "router-default-7f48d5849b-h27lv" (UID: "fdd6746a-7957-4de4-8241-4c1d740f7814") : configmap references non-existent config key: service-ca.crt Sep 10 16:02:16.980524 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.980428 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgdx7\" (UniqueName: \"kubernetes.io/projected/b0e776dd-a8d0-4527-b511-729ca90900bb-kube-api-access-mgdx7\") pod \"ingress-canary-pk2xn\" (UID: \"b0e776dd-a8d0-4527-b511-729ca90900bb\") " pod="openshift-ingress-canary/ingress-canary-pk2xn" Sep 10 16:02:16.980524 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.980499 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b-config\") pod \"console-operator-85b454874b-bt6g6\" (UID: \"70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b\") " pod="openshift-console-operator/console-operator-85b454874b-bt6g6" Sep 10 16:02:16.980918 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.980565 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/befb7be9-0852-4486-a536-219bb03d51f6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-4nbnm\" (UID: \"befb7be9-0852-4486-a536-219bb03d51f6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm" Sep 10 16:02:16.980918 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.980600 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpdtv\" (UniqueName: \"kubernetes.io/projected/70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b-kube-api-access-kpdtv\") pod \"console-operator-85b454874b-bt6g6\" (UID: \"70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b\") " pod="openshift-console-operator/console-operator-85b454874b-bt6g6" Sep 10 16:02:16.980918 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:16.980701 2541 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 16:02:16.980918 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:16.980756 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/befb7be9-0852-4486-a536-219bb03d51f6-cluster-monitoring-operator-tls podName:befb7be9-0852-4486-a536-219bb03d51f6 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:17.480744409 +0000 UTC m=+37.003989584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/befb7be9-0852-4486-a536-219bb03d51f6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-4nbnm" (UID: "befb7be9-0852-4486-a536-219bb03d51f6") : secret "cluster-monitoring-operator-tls" not found Sep 10 16:02:16.980918 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.980700 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39f061d0-48b5-40f0-8428-34d2a6bcc612-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sl7bk\" (UID: \"39f061d0-48b5-40f0-8428-34d2a6bcc612\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sl7bk" Sep 10 16:02:16.980918 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:16.980769 2541 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 16:02:16.980918 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.980797 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txgp4\" (UniqueName: \"kubernetes.io/projected/39f061d0-48b5-40f0-8428-34d2a6bcc612-kube-api-access-txgp4\") pod \"cluster-samples-operator-7cdcfcf76c-sl7bk\" (UID: \"39f061d0-48b5-40f0-8428-34d2a6bcc612\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sl7bk" Sep 10 16:02:16.980918 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.980828 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0e776dd-a8d0-4527-b511-729ca90900bb-cert\") pod \"ingress-canary-pk2xn\" (UID: \"b0e776dd-a8d0-4527-b511-729ca90900bb\") " pod="openshift-ingress-canary/ingress-canary-pk2xn" Sep 10 16:02:16.980918 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.980849 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78552\" (UniqueName: \"kubernetes.io/projected/3a5e2ec6-0e23-4d13-a153-25d193930e01-kube-api-access-78552\") pod \"insights-operator-76bcd6ccf5-89xqs\" (UID: \"3a5e2ec6-0e23-4d13-a153-25d193930e01\") " pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" Sep 10 16:02:16.980918 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.980865 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b-trusted-ca\") pod \"console-operator-85b454874b-bt6g6\" (UID: \"70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b\") " pod="openshift-console-operator/console-operator-85b454874b-bt6g6" Sep 10 16:02:16.980918 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.980897 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b-serving-cert\") pod \"console-operator-85b454874b-bt6g6\" (UID: \"70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b\") " pod="openshift-console-operator/console-operator-85b454874b-bt6g6" Sep 10 16:02:16.981431 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.980931 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-metrics-certs\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:16.981431 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:16.980947 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f061d0-48b5-40f0-8428-34d2a6bcc612-samples-operator-tls podName:39f061d0-48b5-40f0-8428-34d2a6bcc612 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:17.480930016 +0000 UTC m=+37.004175193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/39f061d0-48b5-40f0-8428-34d2a6bcc612-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-sl7bk" (UID: "39f061d0-48b5-40f0-8428-34d2a6bcc612") : secret "samples-operator-tls" not found Sep 10 16:02:16.981431 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.980988 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a5e2ec6-0e23-4d13-a153-25d193930e01-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-89xqs\" (UID: \"3a5e2ec6-0e23-4d13-a153-25d193930e01\") " pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" Sep 10 16:02:16.981431 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.981012 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-stats-auth\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:16.981431 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:16.980993 2541 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 16:02:16.981431 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:16.981065 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-metrics-certs podName:fdd6746a-7957-4de4-8241-4c1d740f7814 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:17.481053571 +0000 UTC m=+37.004298745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-metrics-certs") pod "router-default-7f48d5849b-h27lv" (UID: "fdd6746a-7957-4de4-8241-4c1d740f7814") : secret "router-metrics-certs-default" not found Sep 10 16:02:16.981431 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.981102 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3a5e2ec6-0e23-4d13-a153-25d193930e01-snapshots\") pod \"insights-operator-76bcd6ccf5-89xqs\" (UID: \"3a5e2ec6-0e23-4d13-a153-25d193930e01\") " pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" Sep 10 16:02:16.981431 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.981271 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b-config\") pod \"console-operator-85b454874b-bt6g6\" (UID: \"70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b\") " pod="openshift-console-operator/console-operator-85b454874b-bt6g6" Sep 10 16:02:16.981431 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.981360 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a5e2ec6-0e23-4d13-a153-25d193930e01-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-89xqs\" (UID: \"3a5e2ec6-0e23-4d13-a153-25d193930e01\") " pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" Sep 10 16:02:16.981431 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.981397 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-metrics-tls\") pod \"dns-default-6r7v6\" (UID: \"0c3d3154-d3b2-4886-9bb6-0036bc9bdd35\") " pod="openshift-dns/dns-default-6r7v6" Sep 10 16:02:16.981952 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.981445 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-default-certificate\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:16.981952 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.981476 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a5e2ec6-0e23-4d13-a153-25d193930e01-tmp\") pod \"insights-operator-76bcd6ccf5-89xqs\" (UID: \"3a5e2ec6-0e23-4d13-a153-25d193930e01\") " pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" Sep 10 16:02:16.981952 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.981502 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgqmp\" (UniqueName: \"kubernetes.io/projected/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-kube-api-access-lgqmp\") pod \"dns-default-6r7v6\" (UID: \"0c3d3154-d3b2-4886-9bb6-0036bc9bdd35\") " pod="openshift-dns/dns-default-6r7v6" Sep 10 16:02:16.981952 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.981533 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/befb7be9-0852-4486-a536-219bb03d51f6-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-4nbnm\" (UID: \"befb7be9-0852-4486-a536-219bb03d51f6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm" Sep 10 16:02:16.981952 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.981604 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a5e2ec6-0e23-4d13-a153-25d193930e01-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-89xqs\" (UID: \"3a5e2ec6-0e23-4d13-a153-25d193930e01\") " pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" Sep 10 16:02:16.981952 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.981914 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a5e2ec6-0e23-4d13-a153-25d193930e01-tmp\") pod \"insights-operator-76bcd6ccf5-89xqs\" (UID: \"3a5e2ec6-0e23-4d13-a153-25d193930e01\") " pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" Sep 10 16:02:16.981952 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.981924 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b-trusted-ca\") pod \"console-operator-85b454874b-bt6g6\" (UID: \"70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b\") " pod="openshift-console-operator/console-operator-85b454874b-bt6g6" Sep 10 16:02:16.982264 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.982148 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a5e2ec6-0e23-4d13-a153-25d193930e01-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-89xqs\" (UID: \"3a5e2ec6-0e23-4d13-a153-25d193930e01\") " pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" Sep 10 16:02:16.982339 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.982316 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/befb7be9-0852-4486-a536-219bb03d51f6-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-4nbnm\" (UID: \"befb7be9-0852-4486-a536-219bb03d51f6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm" Sep 10 16:02:16.983175 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.983149 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a5e2ec6-0e23-4d13-a153-25d193930e01-serving-cert\") pod \"insights-operator-76bcd6ccf5-89xqs\" (UID: \"3a5e2ec6-0e23-4d13-a153-25d193930e01\") " pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" Sep 10 16:02:16.984047 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.984023 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b-serving-cert\") pod \"console-operator-85b454874b-bt6g6\" (UID: \"70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b\") " pod="openshift-console-operator/console-operator-85b454874b-bt6g6" Sep 10 16:02:16.984047 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.984037 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-stats-auth\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:16.984168 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.984155 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-default-certificate\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:16.993863 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.993830 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8npjc\" (UniqueName: \"kubernetes.io/projected/befb7be9-0852-4486-a536-219bb03d51f6-kube-api-access-8npjc\") pod \"cluster-monitoring-operator-6cdcc96965-4nbnm\" (UID: \"befb7be9-0852-4486-a536-219bb03d51f6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm" Sep 10 16:02:16.995381 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.995355 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:16.995628 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.995610 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6ldl\" (UniqueName: \"kubernetes.io/projected/fdd6746a-7957-4de4-8241-4c1d740f7814-kube-api-access-b6ldl\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:16.995724 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.995705 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78552\" (UniqueName: \"kubernetes.io/projected/3a5e2ec6-0e23-4d13-a153-25d193930e01-kube-api-access-78552\") pod \"insights-operator-76bcd6ccf5-89xqs\" (UID: \"3a5e2ec6-0e23-4d13-a153-25d193930e01\") " pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" Sep 10 16:02:16.995990 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.995974 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpdtv\" (UniqueName: \"kubernetes.io/projected/70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b-kube-api-access-kpdtv\") pod \"console-operator-85b454874b-bt6g6\" (UID: \"70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b\") " pod="openshift-console-operator/console-operator-85b454874b-bt6g6" Sep 10 16:02:16.997291 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:16.997273 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txgp4\" (UniqueName: \"kubernetes.io/projected/39f061d0-48b5-40f0-8428-34d2a6bcc612-kube-api-access-txgp4\") pod \"cluster-samples-operator-7cdcfcf76c-sl7bk\" (UID: \"39f061d0-48b5-40f0-8428-34d2a6bcc612\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sl7bk" Sep 10 16:02:17.038526 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.038483 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-scdqf" Sep 10 16:02:17.046684 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.046653 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-bt6g6" Sep 10 16:02:17.082200 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.082164 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-config-volume\") pod \"dns-default-6r7v6\" (UID: \"0c3d3154-d3b2-4886-9bb6-0036bc9bdd35\") " pod="openshift-dns/dns-default-6r7v6" Sep 10 16:02:17.082289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.082219 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgdx7\" (UniqueName: \"kubernetes.io/projected/b0e776dd-a8d0-4527-b511-729ca90900bb-kube-api-access-mgdx7\") pod \"ingress-canary-pk2xn\" (UID: \"b0e776dd-a8d0-4527-b511-729ca90900bb\") " pod="openshift-ingress-canary/ingress-canary-pk2xn" Sep 10 16:02:17.082289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.082275 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0e776dd-a8d0-4527-b511-729ca90900bb-cert\") pod \"ingress-canary-pk2xn\" (UID: \"b0e776dd-a8d0-4527-b511-729ca90900bb\") " pod="openshift-ingress-canary/ingress-canary-pk2xn" Sep 10 16:02:17.082393 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.082378 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-metrics-tls\") pod \"dns-default-6r7v6\" (UID: \"0c3d3154-d3b2-4886-9bb6-0036bc9bdd35\") " pod="openshift-dns/dns-default-6r7v6" Sep 10 16:02:17.082434 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.082424 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgqmp\" (UniqueName: \"kubernetes.io/projected/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-kube-api-access-lgqmp\") pod \"dns-default-6r7v6\" (UID: \"0c3d3154-d3b2-4886-9bb6-0036bc9bdd35\") " pod="openshift-dns/dns-default-6r7v6" Sep 10 16:02:17.082484 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.082468 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-tmp-dir\") pod \"dns-default-6r7v6\" (UID: \"0c3d3154-d3b2-4886-9bb6-0036bc9bdd35\") " pod="openshift-dns/dns-default-6r7v6" Sep 10 16:02:17.082546 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:17.082523 2541 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 16:02:17.082606 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:17.082592 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-metrics-tls podName:0c3d3154-d3b2-4886-9bb6-0036bc9bdd35 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:17.582572324 +0000 UTC m=+37.105817503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-metrics-tls") pod "dns-default-6r7v6" (UID: "0c3d3154-d3b2-4886-9bb6-0036bc9bdd35") : secret "dns-default-metrics-tls" not found Sep 10 16:02:17.082779 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.082758 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-tmp-dir\") pod \"dns-default-6r7v6\" (UID: \"0c3d3154-d3b2-4886-9bb6-0036bc9bdd35\") " pod="openshift-dns/dns-default-6r7v6" Sep 10 16:02:17.082854 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.082817 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-config-volume\") pod \"dns-default-6r7v6\" (UID: \"0c3d3154-d3b2-4886-9bb6-0036bc9bdd35\") " pod="openshift-dns/dns-default-6r7v6" Sep 10 16:02:17.084988 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.084969 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0e776dd-a8d0-4527-b511-729ca90900bb-cert\") pod \"ingress-canary-pk2xn\" (UID: \"b0e776dd-a8d0-4527-b511-729ca90900bb\") " pod="openshift-ingress-canary/ingress-canary-pk2xn" Sep 10 16:02:17.092807 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.092637 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgqmp\" (UniqueName: \"kubernetes.io/projected/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-kube-api-access-lgqmp\") pod \"dns-default-6r7v6\" (UID: \"0c3d3154-d3b2-4886-9bb6-0036bc9bdd35\") " pod="openshift-dns/dns-default-6r7v6" Sep 10 16:02:17.092982 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.092959 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgdx7\" (UniqueName: \"kubernetes.io/projected/b0e776dd-a8d0-4527-b511-729ca90900bb-kube-api-access-mgdx7\") pod \"ingress-canary-pk2xn\" (UID: \"b0e776dd-a8d0-4527-b511-729ca90900bb\") " pod="openshift-ingress-canary/ingress-canary-pk2xn" Sep 10 16:02:17.123707 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.123672 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" Sep 10 16:02:17.140860 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.140037 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pk2xn" Sep 10 16:02:17.167988 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.167958 2541 generic.go:356] "Generic (PLEG): container finished" podID="54359979-f90b-4674-b39d-122f7c847055" containerID="2d71e011a72443db4d71bbafc51dfb749ea46f502be5c7e8a6613d398a34ab25" exitCode=0 Sep 10 16:02:17.168120 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.168004 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9crl" event={"ID":"54359979-f90b-4674-b39d-122f7c847055","Type":"ContainerDied","Data":"2d71e011a72443db4d71bbafc51dfb749ea46f502be5c7e8a6613d398a34ab25"} Sep 10 16:02:17.275206 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.274860 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-xbqb7"] Sep 10 16:02:17.310457 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.310431 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xbqb7" Sep 10 16:02:17.312903 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.312875 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-86fxw\"" Sep 10 16:02:17.381708 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.381649 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-scdqf"] Sep 10 16:02:17.382345 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.382290 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:17.383000 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.382943 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-bt6g6"] Sep 10 16:02:17.385494 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.385458 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwnhb\" (UniqueName: \"kubernetes.io/projected/ac6acd6c-a24c-47cc-a461-778f88278b3c-kube-api-access-wwnhb\") pod \"node-resolver-xbqb7\" (UID: \"ac6acd6c-a24c-47cc-a461-778f88278b3c\") " pod="openshift-dns/node-resolver-xbqb7" Sep 10 16:02:17.385607 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.385539 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6acd6c-a24c-47cc-a461-778f88278b3c-tmp-dir\") pod \"node-resolver-xbqb7\" (UID: \"ac6acd6c-a24c-47cc-a461-778f88278b3c\") " pod="openshift-dns/node-resolver-xbqb7" Sep 10 16:02:17.385667 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.385623 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:17.385667 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.385650 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ac6acd6c-a24c-47cc-a461-778f88278b3c-hosts-file\") pod \"node-resolver-xbqb7\" (UID: \"ac6acd6c-a24c-47cc-a461-778f88278b3c\") " pod="openshift-dns/node-resolver-xbqb7" Sep 10 16:02:17.385780 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:17.385762 2541 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 16:02:17.385816 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:17.385781 2541 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65b9595f8c-v7mjc: secret "image-registry-tls" not found Sep 10 16:02:17.385846 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:17.385829 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls podName:13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad nodeName:}" failed. No retries permitted until 2026-09-10 16:02:18.385814189 +0000 UTC m=+37.909059369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls") pod "image-registry-65b9595f8c-v7mjc" (UID: "13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad") : secret "image-registry-tls" not found Sep 10 16:02:17.459943 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.459916 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:17.486070 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.486036 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-metrics-certs\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:17.486225 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.486101 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ac6acd6c-a24c-47cc-a461-778f88278b3c-hosts-file\") pod \"node-resolver-xbqb7\" (UID: \"ac6acd6c-a24c-47cc-a461-778f88278b3c\") " pod="openshift-dns/node-resolver-xbqb7" Sep 10 16:02:17.486225 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.486160 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdd6746a-7957-4de4-8241-4c1d740f7814-service-ca-bundle\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:17.486225 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:17.486178 2541 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 16:02:17.486225 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.486191 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwnhb\" (UniqueName: \"kubernetes.io/projected/ac6acd6c-a24c-47cc-a461-778f88278b3c-kube-api-access-wwnhb\") pod \"node-resolver-xbqb7\" (UID: \"ac6acd6c-a24c-47cc-a461-778f88278b3c\") " pod="openshift-dns/node-resolver-xbqb7" Sep 10 16:02:17.486225 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.486221 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/befb7be9-0852-4486-a536-219bb03d51f6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-4nbnm\" (UID: \"befb7be9-0852-4486-a536-219bb03d51f6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm" Sep 10 16:02:17.486474 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:17.486241 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-metrics-certs podName:fdd6746a-7957-4de4-8241-4c1d740f7814 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:18.486225116 +0000 UTC m=+38.009470290 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-metrics-certs") pod "router-default-7f48d5849b-h27lv" (UID: "fdd6746a-7957-4de4-8241-4c1d740f7814") : secret "router-metrics-certs-default" not found Sep 10 16:02:17.486474 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.486283 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39f061d0-48b5-40f0-8428-34d2a6bcc612-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sl7bk\" (UID: \"39f061d0-48b5-40f0-8428-34d2a6bcc612\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sl7bk" Sep 10 16:02:17.486474 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:17.486326 2541 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 16:02:17.486474 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.486336 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6acd6c-a24c-47cc-a461-778f88278b3c-tmp-dir\") pod \"node-resolver-xbqb7\" (UID: \"ac6acd6c-a24c-47cc-a461-778f88278b3c\") " pod="openshift-dns/node-resolver-xbqb7" Sep 10 16:02:17.486474 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:17.486365 2541 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 16:02:17.486474 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:17.486377 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/befb7be9-0852-4486-a536-219bb03d51f6-cluster-monitoring-operator-tls podName:befb7be9-0852-4486-a536-219bb03d51f6 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:18.486360338 +0000 UTC m=+38.009605525 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/befb7be9-0852-4486-a536-219bb03d51f6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-4nbnm" (UID: "befb7be9-0852-4486-a536-219bb03d51f6") : secret "cluster-monitoring-operator-tls" not found Sep 10 16:02:17.486474 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.486288 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ac6acd6c-a24c-47cc-a461-778f88278b3c-hosts-file\") pod \"node-resolver-xbqb7\" (UID: \"ac6acd6c-a24c-47cc-a461-778f88278b3c\") " pod="openshift-dns/node-resolver-xbqb7" Sep 10 16:02:17.486474 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:17.486403 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdd6746a-7957-4de4-8241-4c1d740f7814-service-ca-bundle podName:fdd6746a-7957-4de4-8241-4c1d740f7814 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:18.486386109 +0000 UTC m=+38.009631297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fdd6746a-7957-4de4-8241-4c1d740f7814-service-ca-bundle") pod "router-default-7f48d5849b-h27lv" (UID: "fdd6746a-7957-4de4-8241-4c1d740f7814") : configmap references non-existent config key: service-ca.crt Sep 10 16:02:17.486474 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:17.486432 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f061d0-48b5-40f0-8428-34d2a6bcc612-samples-operator-tls podName:39f061d0-48b5-40f0-8428-34d2a6bcc612 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:18.486422156 +0000 UTC m=+38.009667337 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/39f061d0-48b5-40f0-8428-34d2a6bcc612-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-sl7bk" (UID: "39f061d0-48b5-40f0-8428-34d2a6bcc612") : secret "samples-operator-tls" not found Sep 10 16:02:17.486754 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.486585 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6acd6c-a24c-47cc-a461-778f88278b3c-tmp-dir\") pod \"node-resolver-xbqb7\" (UID: \"ac6acd6c-a24c-47cc-a461-778f88278b3c\") " pod="openshift-dns/node-resolver-xbqb7" Sep 10 16:02:17.503371 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.503342 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwnhb\" (UniqueName: \"kubernetes.io/projected/ac6acd6c-a24c-47cc-a461-778f88278b3c-kube-api-access-wwnhb\") pod \"node-resolver-xbqb7\" (UID: \"ac6acd6c-a24c-47cc-a461-778f88278b3c\") " pod="openshift-dns/node-resolver-xbqb7" Sep 10 16:02:17.511206 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.511178 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pk2xn"] Sep 10 16:02:17.515553 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.515523 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-89xqs"] Sep 10 16:02:17.521168 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:02:17.521133 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5e2ec6_0e23_4d13_a153_25d193930e01.slice/crio-e3663bb8fbaf505597d8a1c3cff53e2ce7bd519d8ebce1062bdb7e579022265f WatchSource:0}: Error finding container e3663bb8fbaf505597d8a1c3cff53e2ce7bd519d8ebce1062bdb7e579022265f: Status 404 returned error can't find the container with id e3663bb8fbaf505597d8a1c3cff53e2ce7bd519d8ebce1062bdb7e579022265f Sep 10 16:02:17.523036 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:02:17.523004 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e776dd_a8d0_4527_b511_729ca90900bb.slice/crio-5284b94850eec7516e41389225e8d8fef9846d56bae405f60e0bc7b10d802698 WatchSource:0}: Error finding container 5284b94850eec7516e41389225e8d8fef9846d56bae405f60e0bc7b10d802698: Status 404 returned error can't find the container with id 5284b94850eec7516e41389225e8d8fef9846d56bae405f60e0bc7b10d802698 Sep 10 16:02:17.523640 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:02:17.523614 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b8f4ef_c89f_48fe_b006_de3f7f6d6d3b.slice/crio-e7b3a81781e468eefb1ecb8c50c4f5da11f16a397827afc57d08d2ac8c6fbf9f WatchSource:0}: Error finding container e7b3a81781e468eefb1ecb8c50c4f5da11f16a397827afc57d08d2ac8c6fbf9f: Status 404 returned error can't find the container with id e7b3a81781e468eefb1ecb8c50c4f5da11f16a397827afc57d08d2ac8c6fbf9f Sep 10 16:02:17.558829 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.558795 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:17.587038 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.587006 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-metrics-tls\") pod \"dns-default-6r7v6\" (UID: \"0c3d3154-d3b2-4886-9bb6-0036bc9bdd35\") " pod="openshift-dns/dns-default-6r7v6" Sep 10 16:02:17.587175 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:17.587155 2541 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 16:02:17.587232 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:17.587224 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-metrics-tls podName:0c3d3154-d3b2-4886-9bb6-0036bc9bdd35 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:18.587207636 +0000 UTC m=+38.110452810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-metrics-tls") pod "dns-default-6r7v6" (UID: "0c3d3154-d3b2-4886-9bb6-0036bc9bdd35") : secret "dns-default-metrics-tls" not found Sep 10 16:02:17.619270 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.619197 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xbqb7" Sep 10 16:02:17.655574 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:02:17.655543 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6acd6c_a24c_47cc_a461_778f88278b3c.slice/crio-07fd31a11e5286a5850cbfbe10fd4fbf4691c9ac91a4783aa0b50fa8f1dc873e WatchSource:0}: Error finding container 07fd31a11e5286a5850cbfbe10fd4fbf4691c9ac91a4783aa0b50fa8f1dc873e: Status 404 returned error can't find the container with id 07fd31a11e5286a5850cbfbe10fd4fbf4691c9ac91a4783aa0b50fa8f1dc873e Sep 10 16:02:17.664074 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.664047 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:17.974192 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.973959 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:02:17.974192 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.974016 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:02:17.974451 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.974022 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:02:17.976770 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.976739 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g5gt7\"" Sep 10 16:02:17.976900 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.976856 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 16:02:17.976978 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.976946 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 16:02:17.977031 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.976744 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 16:02:17.977182 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.977164 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-swxfx\"" Sep 10 16:02:17.977275 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:17.977258 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 16:02:18.173044 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:18.173004 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-scdqf" event={"ID":"ae665a3c-336f-4057-8e8e-ca9572f1fd58","Type":"ContainerStarted","Data":"ce0ff55415a2a4d3e7c315105eb1caa50a1b2f27620d7c98c2389a121fe0611b"} Sep 10 16:02:18.176523 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:18.175671 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-bt6g6" event={"ID":"70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b","Type":"ContainerStarted","Data":"e7b3a81781e468eefb1ecb8c50c4f5da11f16a397827afc57d08d2ac8c6fbf9f"} Sep 10 16:02:18.177531 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:18.177499 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" event={"ID":"3a5e2ec6-0e23-4d13-a153-25d193930e01","Type":"ContainerStarted","Data":"e3663bb8fbaf505597d8a1c3cff53e2ce7bd519d8ebce1062bdb7e579022265f"} Sep 10 16:02:18.180542 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:18.180509 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xbqb7" event={"ID":"ac6acd6c-a24c-47cc-a461-778f88278b3c","Type":"ContainerStarted","Data":"f3b410dec63a0582c4d71d0038cd36788748240d600349edfaa800c6a1275e95"} Sep 10 16:02:18.180662 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:18.180563 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xbqb7" event={"ID":"ac6acd6c-a24c-47cc-a461-778f88278b3c","Type":"ContainerStarted","Data":"07fd31a11e5286a5850cbfbe10fd4fbf4691c9ac91a4783aa0b50fa8f1dc873e"} Sep 10 16:02:18.187565 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:18.187534 2541 generic.go:356] "Generic (PLEG): container finished" podID="54359979-f90b-4674-b39d-122f7c847055" containerID="9deac25785831d38a301221707c4da936847fafbb3c3eba9f19612ae4d14e42f" exitCode=0 Sep 10 16:02:18.187685 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:18.187630 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9crl" event={"ID":"54359979-f90b-4674-b39d-122f7c847055","Type":"ContainerDied","Data":"9deac25785831d38a301221707c4da936847fafbb3c3eba9f19612ae4d14e42f"} Sep 10 16:02:18.190383 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:18.190356 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pk2xn" event={"ID":"b0e776dd-a8d0-4527-b511-729ca90900bb","Type":"ContainerStarted","Data":"5284b94850eec7516e41389225e8d8fef9846d56bae405f60e0bc7b10d802698"} Sep 10 16:02:18.198720 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:18.198365 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xbqb7" podStartSLOduration=1.19834683 podStartE2EDuration="1.19834683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:18.197341051 +0000 UTC m=+37.720586248" watchObservedRunningTime="2026-09-10 16:02:18.19834683 +0000 UTC m=+37.721592027" Sep 10 16:02:18.266728 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:18.265563 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:18.395494 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:18.395290 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:18.395643 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:18.395517 2541 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 16:02:18.395643 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:18.395535 2541 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65b9595f8c-v7mjc: secret "image-registry-tls" not found Sep 10 16:02:18.395643 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:18.395592 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls podName:13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad nodeName:}" failed. No retries permitted until 2026-09-10 16:02:20.395574037 +0000 UTC m=+39.918819225 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls") pod "image-registry-65b9595f8c-v7mjc" (UID: "13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad") : secret "image-registry-tls" not found Sep 10 16:02:18.497325 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:18.496128 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/befb7be9-0852-4486-a536-219bb03d51f6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-4nbnm\" (UID: \"befb7be9-0852-4486-a536-219bb03d51f6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm" Sep 10 16:02:18.497325 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:18.496197 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39f061d0-48b5-40f0-8428-34d2a6bcc612-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sl7bk\" (UID: \"39f061d0-48b5-40f0-8428-34d2a6bcc612\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sl7bk" Sep 10 16:02:18.497325 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:18.496240 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-metrics-certs\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:18.497325 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:18.496394 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdd6746a-7957-4de4-8241-4c1d740f7814-service-ca-bundle\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:18.497325 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:18.496554 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdd6746a-7957-4de4-8241-4c1d740f7814-service-ca-bundle podName:fdd6746a-7957-4de4-8241-4c1d740f7814 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:20.496534691 +0000 UTC m=+40.019779886 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fdd6746a-7957-4de4-8241-4c1d740f7814-service-ca-bundle") pod "router-default-7f48d5849b-h27lv" (UID: "fdd6746a-7957-4de4-8241-4c1d740f7814") : configmap references non-existent config key: service-ca.crt Sep 10 16:02:18.497325 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:18.496989 2541 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 16:02:18.497325 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:18.497044 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/befb7be9-0852-4486-a536-219bb03d51f6-cluster-monitoring-operator-tls podName:befb7be9-0852-4486-a536-219bb03d51f6 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:20.497027794 +0000 UTC m=+40.020272986 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/befb7be9-0852-4486-a536-219bb03d51f6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-4nbnm" (UID: "befb7be9-0852-4486-a536-219bb03d51f6") : secret "cluster-monitoring-operator-tls" not found Sep 10 16:02:18.497325 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:18.497230 2541 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 16:02:18.497325 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:18.497275 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-metrics-certs podName:fdd6746a-7957-4de4-8241-4c1d740f7814 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:20.497261544 +0000 UTC m=+40.020506722 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-metrics-certs") pod "router-default-7f48d5849b-h27lv" (UID: "fdd6746a-7957-4de4-8241-4c1d740f7814") : secret "router-metrics-certs-default" not found Sep 10 16:02:18.498029 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:18.497950 2541 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 16:02:18.498029 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:18.498010 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f061d0-48b5-40f0-8428-34d2a6bcc612-samples-operator-tls podName:39f061d0-48b5-40f0-8428-34d2a6bcc612 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:20.497994979 +0000 UTC m=+40.021240162 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/39f061d0-48b5-40f0-8428-34d2a6bcc612-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-sl7bk" (UID: "39f061d0-48b5-40f0-8428-34d2a6bcc612") : secret "samples-operator-tls" not found Sep 10 16:02:18.597937 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:18.597658 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-metrics-tls\") pod \"dns-default-6r7v6\" (UID: \"0c3d3154-d3b2-4886-9bb6-0036bc9bdd35\") " pod="openshift-dns/dns-default-6r7v6" Sep 10 16:02:18.597937 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:18.597879 2541 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 16:02:18.598146 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:18.597958 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-metrics-tls podName:0c3d3154-d3b2-4886-9bb6-0036bc9bdd35 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:20.59793528 +0000 UTC m=+40.121180479 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-metrics-tls") pod "dns-default-6r7v6" (UID: "0c3d3154-d3b2-4886-9bb6-0036bc9bdd35") : secret "dns-default-metrics-tls" not found Sep 10 16:02:19.200224 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:19.200181 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9crl" event={"ID":"54359979-f90b-4674-b39d-122f7c847055","Type":"ContainerStarted","Data":"502301bb46a9b6a0e9ea66cd88f39b4e98ad3011b7cdfb7c1e83fbad577dccc8"} Sep 10 16:02:19.220604 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:19.220498 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-l9crl" podStartSLOduration=8.600260914 podStartE2EDuration="37.220482014s" totalImagesPullingTime="28.6202211s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:01:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:19.220047609 +0000 UTC m=+38.743292807" watchObservedRunningTime="2026-09-10 16:02:19.220482014 +0000 UTC m=+38.743727211" Sep 10 16:02:19.344486 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:19.344456 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:19.446407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:19.444969 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:20.415836 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:20.415801 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:20.416259 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:20.415968 2541 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 16:02:20.416259 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:20.415986 2541 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65b9595f8c-v7mjc: secret "image-registry-tls" not found Sep 10 16:02:20.416259 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:20.416049 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls podName:13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad nodeName:}" failed. No retries permitted until 2026-09-10 16:02:24.41603115 +0000 UTC m=+43.939276336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls") pod "image-registry-65b9595f8c-v7mjc" (UID: "13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad") : secret "image-registry-tls" not found Sep 10 16:02:20.516730 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:20.516692 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdd6746a-7957-4de4-8241-4c1d740f7814-service-ca-bundle\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:20.516884 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:20.516742 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/befb7be9-0852-4486-a536-219bb03d51f6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-4nbnm\" (UID: \"befb7be9-0852-4486-a536-219bb03d51f6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm" Sep 10 16:02:20.516884 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:20.516766 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39f061d0-48b5-40f0-8428-34d2a6bcc612-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sl7bk\" (UID: \"39f061d0-48b5-40f0-8428-34d2a6bcc612\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sl7bk" Sep 10 16:02:20.516884 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:20.516806 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-metrics-certs\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:20.517007 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:20.516878 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdd6746a-7957-4de4-8241-4c1d740f7814-service-ca-bundle podName:fdd6746a-7957-4de4-8241-4c1d740f7814 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:24.516857093 +0000 UTC m=+44.040102291 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fdd6746a-7957-4de4-8241-4c1d740f7814-service-ca-bundle") pod "router-default-7f48d5849b-h27lv" (UID: "fdd6746a-7957-4de4-8241-4c1d740f7814") : configmap references non-existent config key: service-ca.crt Sep 10 16:02:20.517007 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:20.516905 2541 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 16:02:20.517007 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:20.516905 2541 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 16:02:20.517007 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:20.516878 2541 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 16:02:20.517007 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:20.516958 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-metrics-certs podName:fdd6746a-7957-4de4-8241-4c1d740f7814 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:24.516943416 +0000 UTC m=+44.040188605 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-metrics-certs") pod "router-default-7f48d5849b-h27lv" (UID: "fdd6746a-7957-4de4-8241-4c1d740f7814") : secret "router-metrics-certs-default" not found Sep 10 16:02:20.517007 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:20.516975 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f061d0-48b5-40f0-8428-34d2a6bcc612-samples-operator-tls podName:39f061d0-48b5-40f0-8428-34d2a6bcc612 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:24.516968597 +0000 UTC m=+44.040213772 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/39f061d0-48b5-40f0-8428-34d2a6bcc612-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-sl7bk" (UID: "39f061d0-48b5-40f0-8428-34d2a6bcc612") : secret "samples-operator-tls" not found Sep 10 16:02:20.517007 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:20.516998 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/befb7be9-0852-4486-a536-219bb03d51f6-cluster-monitoring-operator-tls podName:befb7be9-0852-4486-a536-219bb03d51f6 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:24.516984938 +0000 UTC m=+44.040230117 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/befb7be9-0852-4486-a536-219bb03d51f6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-4nbnm" (UID: "befb7be9-0852-4486-a536-219bb03d51f6") : secret "cluster-monitoring-operator-tls" not found Sep 10 16:02:20.618246 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:20.618197 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-metrics-tls\") pod \"dns-default-6r7v6\" (UID: \"0c3d3154-d3b2-4886-9bb6-0036bc9bdd35\") " pod="openshift-dns/dns-default-6r7v6" Sep 10 16:02:20.618465 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:20.618352 2541 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 16:02:20.618465 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:20.618415 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-metrics-tls podName:0c3d3154-d3b2-4886-9bb6-0036bc9bdd35 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:24.618400704 +0000 UTC m=+44.141645878 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-metrics-tls") pod "dns-default-6r7v6" (UID: "0c3d3154-d3b2-4886-9bb6-0036bc9bdd35") : secret "dns-default-metrics-tls" not found Sep 10 16:02:20.767289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:20.767173 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:20.828178 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:20.828136 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:22.211281 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:22.211202 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pk2xn" event={"ID":"b0e776dd-a8d0-4527-b511-729ca90900bb","Type":"ContainerStarted","Data":"2857d499fa309793d13d0a5f5cd2efac048006ed66936ba61d9a150810dfa1a6"} Sep 10 16:02:22.213650 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:22.213621 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-scdqf" event={"ID":"ae665a3c-336f-4057-8e8e-ca9572f1fd58","Type":"ContainerStarted","Data":"65535bca0b2ba62a2efb8b41e147ddf15b5ffc1221946cc7f7a3d4f260b96758"} Sep 10 16:02:22.215428 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:22.215403 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-bt6g6" event={"ID":"70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b","Type":"ContainerStarted","Data":"fb8754571c8bad9b83169528cea5ee4272fcb55361cbc2e0b13f21187271fe63"} Sep 10 16:02:22.215577 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:22.215557 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-bt6g6" Sep 10 16:02:22.217049 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:22.217011 2541 patch_prober.go:28] interesting pod/console-operator-85b454874b-bt6g6 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.8:8443/readyz\": dial tcp 10.132.0.8:8443: connect: connection refused" start-of-body= Sep 10 16:02:22.217156 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:22.217074 2541 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-85b454874b-bt6g6" podUID="70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.8:8443/readyz\": dial tcp 10.132.0.8:8443: connect: connection refused" Sep 10 16:02:22.219008 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:22.218976 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" event={"ID":"3a5e2ec6-0e23-4d13-a153-25d193930e01","Type":"ContainerStarted","Data":"3fa58b942592bacbb8fa880bddafec80507e18d402e9038be3938744f9d0c72c"} Sep 10 16:02:22.235811 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:22.235752 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pk2xn" podStartSLOduration=1.7981181849999999 podStartE2EDuration="6.23573416s" totalImagesPullingTime="4.437615975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:02:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:22.235427203 +0000 UTC m=+41.758672413" watchObservedRunningTime="2026-09-10 16:02:22.23573416 +0000 UTC m=+41.758979357" Sep 10 16:02:22.254569 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:22.254508 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-bt6g6" podStartSLOduration=1.823878923 podStartE2EDuration="6.254487496s" totalImagesPullingTime="4.430608573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:02:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:22.254276467 +0000 UTC m=+41.777521666" watchObservedRunningTime="2026-09-10 16:02:22.254487496 +0000 UTC m=+41.777732694" Sep 10 16:02:22.286356 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:22.286276 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" podStartSLOduration=2.850483251 podStartE2EDuration="7.286253312s" totalImagesPullingTime="4.435770061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:02:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:22.285287698 +0000 UTC m=+41.808532895" watchObservedRunningTime="2026-09-10 16:02:22.286253312 +0000 UTC m=+41.809498509" Sep 10 16:02:22.309567 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:22.309517 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-scdqf" podStartSLOduration=2.87377336 podStartE2EDuration="7.309498847s" totalImagesPullingTime="4.435725487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:02:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:22.309210297 +0000 UTC m=+41.832455494" watchObservedRunningTime="2026-09-10 16:02:22.309498847 +0000 UTC m=+41.832744036" Sep 10 16:02:22.439428 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:22.439393 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:22.522945 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:22.522860 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:23.013784 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.013751 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:23.116874 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.116835 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:23.226294 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.226271 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/0.log" Sep 10 16:02:23.226678 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.226328 2541 generic.go:356] "Generic (PLEG): container finished" podID="70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b" containerID="fb8754571c8bad9b83169528cea5ee4272fcb55361cbc2e0b13f21187271fe63" exitCode=255 Sep 10 16:02:23.226678 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.226440 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-bt6g6" event={"ID":"70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b","Type":"ContainerDied","Data":"fb8754571c8bad9b83169528cea5ee4272fcb55361cbc2e0b13f21187271fe63"} Sep 10 16:02:23.232225 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.227603 2541 scope.go:117] "RemoveContainer" containerID="fb8754571c8bad9b83169528cea5ee4272fcb55361cbc2e0b13f21187271fe63" Sep 10 16:02:23.265000 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.264934 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ztp4g"] Sep 10 16:02:23.289196 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.289166 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ztp4g"] Sep 10 16:02:23.289377 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.289282 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:23.291414 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.291389 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 16:02:23.291519 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.291400 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6s77g\"" Sep 10 16:02:23.291519 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.291491 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 16:02:23.301129 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.301106 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:23.378109 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.378058 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:23.443894 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.443856 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/946ce162-a05d-4e23-9c7c-5be6a51e0595-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:23.444057 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.443922 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/946ce162-a05d-4e23-9c7c-5be6a51e0595-data-volume\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:23.444057 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.443951 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjj5q\" (UniqueName: \"kubernetes.io/projected/946ce162-a05d-4e23-9c7c-5be6a51e0595-kube-api-access-tjj5q\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:23.444057 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.443991 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/946ce162-a05d-4e23-9c7c-5be6a51e0595-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:23.444057 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.444026 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/946ce162-a05d-4e23-9c7c-5be6a51e0595-crio-socket\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:23.493425 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.493388 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:23.544536 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.544446 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjj5q\" (UniqueName: \"kubernetes.io/projected/946ce162-a05d-4e23-9c7c-5be6a51e0595-kube-api-access-tjj5q\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:23.544536 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.544534 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/946ce162-a05d-4e23-9c7c-5be6a51e0595-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:23.544746 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.544563 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/946ce162-a05d-4e23-9c7c-5be6a51e0595-crio-socket\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:23.544746 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.544662 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/946ce162-a05d-4e23-9c7c-5be6a51e0595-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:23.544746 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.544736 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/946ce162-a05d-4e23-9c7c-5be6a51e0595-data-volume\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:23.544906 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.544736 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/946ce162-a05d-4e23-9c7c-5be6a51e0595-crio-socket\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:23.544906 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:23.544669 2541 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 16:02:23.544906 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:23.544708 2541 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 16:02:23.544906 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:23.544864 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/946ce162-a05d-4e23-9c7c-5be6a51e0595-kube-rbac-proxy-cm podName:946ce162-a05d-4e23-9c7c-5be6a51e0595 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:24.04484478 +0000 UTC m=+43.568089963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/946ce162-a05d-4e23-9c7c-5be6a51e0595-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ztp4g" (UID: "946ce162-a05d-4e23-9c7c-5be6a51e0595") : configmap "kube-rbac-proxy" not found Sep 10 16:02:23.544906 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:23.544885 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/946ce162-a05d-4e23-9c7c-5be6a51e0595-insights-runtime-extractor-tls podName:946ce162-a05d-4e23-9c7c-5be6a51e0595 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:24.044874958 +0000 UTC m=+43.568120137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/946ce162-a05d-4e23-9c7c-5be6a51e0595-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ztp4g" (UID: "946ce162-a05d-4e23-9c7c-5be6a51e0595") : secret "insights-runtime-extractor-tls" not found Sep 10 16:02:23.545148 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.545057 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/946ce162-a05d-4e23-9c7c-5be6a51e0595-data-volume\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:23.556357 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.556326 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjj5q\" (UniqueName: \"kubernetes.io/projected/946ce162-a05d-4e23-9c7c-5be6a51e0595-kube-api-access-tjj5q\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:23.645048 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:23.645012 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:24.048476 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:24.048436 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/946ce162-a05d-4e23-9c7c-5be6a51e0595-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:24.048651 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:24.048573 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/946ce162-a05d-4e23-9c7c-5be6a51e0595-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:24.048651 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:24.048585 2541 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 16:02:24.048651 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:24.048647 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/946ce162-a05d-4e23-9c7c-5be6a51e0595-insights-runtime-extractor-tls podName:946ce162-a05d-4e23-9c7c-5be6a51e0595 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:25.048630383 +0000 UTC m=+44.571875558 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/946ce162-a05d-4e23-9c7c-5be6a51e0595-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ztp4g" (UID: "946ce162-a05d-4e23-9c7c-5be6a51e0595") : secret "insights-runtime-extractor-tls" not found Sep 10 16:02:24.048790 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:24.048657 2541 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 16:02:24.048790 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:24.048722 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/946ce162-a05d-4e23-9c7c-5be6a51e0595-kube-rbac-proxy-cm podName:946ce162-a05d-4e23-9c7c-5be6a51e0595 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:25.048704344 +0000 UTC m=+44.571949536 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/946ce162-a05d-4e23-9c7c-5be6a51e0595-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ztp4g" (UID: "946ce162-a05d-4e23-9c7c-5be6a51e0595") : configmap "kube-rbac-proxy" not found Sep 10 16:02:24.231988 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:24.231963 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/1.log" Sep 10 16:02:24.232499 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:24.232482 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/0.log" Sep 10 16:02:24.232578 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:24.232523 2541 generic.go:356] "Generic (PLEG): container finished" podID="70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b" containerID="bd6d5cea0b409b164a833fc561c49ebf60dec8d4c9059f427fd69a976dab00a8" exitCode=255 Sep 10 16:02:24.232625 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:24.232589 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-bt6g6" event={"ID":"70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b","Type":"ContainerDied","Data":"bd6d5cea0b409b164a833fc561c49ebf60dec8d4c9059f427fd69a976dab00a8"} Sep 10 16:02:24.232680 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:24.232638 2541 scope.go:117] "RemoveContainer" containerID="fb8754571c8bad9b83169528cea5ee4272fcb55361cbc2e0b13f21187271fe63" Sep 10 16:02:24.232984 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:24.232832 2541 scope.go:117] "RemoveContainer" containerID="bd6d5cea0b409b164a833fc561c49ebf60dec8d4c9059f427fd69a976dab00a8" Sep 10 16:02:24.233080 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:24.233021 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-bt6g6_openshift-console-operator(70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b)\"" pod="openshift-console-operator/console-operator-85b454874b-bt6g6" podUID="70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b" Sep 10 16:02:24.452713 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:24.452676 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:24.452886 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:24.452789 2541 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 16:02:24.452886 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:24.452802 2541 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65b9595f8c-v7mjc: secret "image-registry-tls" not found Sep 10 16:02:24.452886 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:24.452851 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls podName:13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad nodeName:}" failed. No retries permitted until 2026-09-10 16:02:32.452838206 +0000 UTC m=+51.976083381 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls") pod "image-registry-65b9595f8c-v7mjc" (UID: "13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad") : secret "image-registry-tls" not found Sep 10 16:02:24.553489 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:24.553453 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/befb7be9-0852-4486-a536-219bb03d51f6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-4nbnm\" (UID: \"befb7be9-0852-4486-a536-219bb03d51f6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm" Sep 10 16:02:24.553489 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:24.553495 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39f061d0-48b5-40f0-8428-34d2a6bcc612-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sl7bk\" (UID: \"39f061d0-48b5-40f0-8428-34d2a6bcc612\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sl7bk" Sep 10 16:02:24.553685 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:24.553524 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-metrics-certs\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:24.553685 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:24.553616 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdd6746a-7957-4de4-8241-4c1d740f7814-service-ca-bundle\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:24.553685 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:24.553618 2541 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 16:02:24.553789 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:24.553677 2541 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 16:02:24.553789 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:24.553626 2541 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 16:02:24.553789 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:24.553695 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/befb7be9-0852-4486-a536-219bb03d51f6-cluster-monitoring-operator-tls podName:befb7be9-0852-4486-a536-219bb03d51f6 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:32.55367555 +0000 UTC m=+52.076920739 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/befb7be9-0852-4486-a536-219bb03d51f6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-4nbnm" (UID: "befb7be9-0852-4486-a536-219bb03d51f6") : secret "cluster-monitoring-operator-tls" not found Sep 10 16:02:24.553789 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:24.553748 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-metrics-certs podName:fdd6746a-7957-4de4-8241-4c1d740f7814 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:32.553737479 +0000 UTC m=+52.076982655 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-metrics-certs") pod "router-default-7f48d5849b-h27lv" (UID: "fdd6746a-7957-4de4-8241-4c1d740f7814") : secret "router-metrics-certs-default" not found Sep 10 16:02:24.553789 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:24.553762 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdd6746a-7957-4de4-8241-4c1d740f7814-service-ca-bundle podName:fdd6746a-7957-4de4-8241-4c1d740f7814 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:32.553754005 +0000 UTC m=+52.076999183 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fdd6746a-7957-4de4-8241-4c1d740f7814-service-ca-bundle") pod "router-default-7f48d5849b-h27lv" (UID: "fdd6746a-7957-4de4-8241-4c1d740f7814") : configmap references non-existent config key: service-ca.crt Sep 10 16:02:24.553789 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:24.553777 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f061d0-48b5-40f0-8428-34d2a6bcc612-samples-operator-tls podName:39f061d0-48b5-40f0-8428-34d2a6bcc612 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:32.553769725 +0000 UTC m=+52.077014900 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/39f061d0-48b5-40f0-8428-34d2a6bcc612-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-sl7bk" (UID: "39f061d0-48b5-40f0-8428-34d2a6bcc612") : secret "samples-operator-tls" not found Sep 10 16:02:24.616209 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:24.616175 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:24.654607 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:24.654568 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-metrics-tls\") pod \"dns-default-6r7v6\" (UID: \"0c3d3154-d3b2-4886-9bb6-0036bc9bdd35\") " pod="openshift-dns/dns-default-6r7v6" Sep 10 16:02:24.654766 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:24.654737 2541 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 16:02:24.654806 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:24.654798 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-metrics-tls podName:0c3d3154-d3b2-4886-9bb6-0036bc9bdd35 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:32.6547818 +0000 UTC m=+52.178026991 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-metrics-tls") pod "dns-default-6r7v6" (UID: "0c3d3154-d3b2-4886-9bb6-0036bc9bdd35") : secret "dns-default-metrics-tls" not found Sep 10 16:02:24.715459 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:24.715382 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:24.810911 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:24.810885 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xbqb7_ac6acd6c-a24c-47cc-a461-778f88278b3c/dns-node-resolver/0.log" Sep 10 16:02:25.058553 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:25.058454 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/946ce162-a05d-4e23-9c7c-5be6a51e0595-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:25.058553 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:25.058552 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/946ce162-a05d-4e23-9c7c-5be6a51e0595-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:25.058795 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:25.058618 2541 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 16:02:25.058795 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:25.058652 2541 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 16:02:25.058795 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:25.058670 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/946ce162-a05d-4e23-9c7c-5be6a51e0595-kube-rbac-proxy-cm podName:946ce162-a05d-4e23-9c7c-5be6a51e0595 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:27.058657455 +0000 UTC m=+46.581902630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/946ce162-a05d-4e23-9c7c-5be6a51e0595-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ztp4g" (UID: "946ce162-a05d-4e23-9c7c-5be6a51e0595") : configmap "kube-rbac-proxy" not found Sep 10 16:02:25.058795 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:25.058721 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/946ce162-a05d-4e23-9c7c-5be6a51e0595-insights-runtime-extractor-tls podName:946ce162-a05d-4e23-9c7c-5be6a51e0595 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:27.058701214 +0000 UTC m=+46.581946402 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/946ce162-a05d-4e23-9c7c-5be6a51e0595-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ztp4g" (UID: "946ce162-a05d-4e23-9c7c-5be6a51e0595") : secret "insights-runtime-extractor-tls" not found Sep 10 16:02:25.237785 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:25.237758 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/1.log" Sep 10 16:02:25.238227 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:25.238187 2541 scope.go:117] "RemoveContainer" containerID="bd6d5cea0b409b164a833fc561c49ebf60dec8d4c9059f427fd69a976dab00a8" Sep 10 16:02:25.238424 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:25.238386 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-bt6g6_openshift-console-operator(70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b)\"" pod="openshift-console-operator/console-operator-85b454874b-bt6g6" podUID="70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b" Sep 10 16:02:26.018850 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:26.018818 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rxmz9_7e5689cb-d3f5-4d8c-b91c-a0e40aeeeef9/node-ca/0.log" Sep 10 16:02:26.207432 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:26.207399 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:26.291270 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:26.291185 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:26.392258 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:26.392228 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:26.410754 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:26.410730 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pk2xn_b0e776dd-a8d0-4527-b511-729ca90900bb/serve-healthcheck-canary/0.log" Sep 10 16:02:27.047801 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:27.047766 2541 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-bt6g6" Sep 10 16:02:27.048153 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:27.048141 2541 scope.go:117] "RemoveContainer" containerID="bd6d5cea0b409b164a833fc561c49ebf60dec8d4c9059f427fd69a976dab00a8" Sep 10 16:02:27.048350 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:27.048328 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-bt6g6_openshift-console-operator(70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b)\"" pod="openshift-console-operator/console-operator-85b454874b-bt6g6" podUID="70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b" Sep 10 16:02:27.076235 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:27.076196 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/946ce162-a05d-4e23-9c7c-5be6a51e0595-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:27.076422 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:27.076330 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/946ce162-a05d-4e23-9c7c-5be6a51e0595-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:27.076422 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:27.076340 2541 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 16:02:27.076422 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:27.076399 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/946ce162-a05d-4e23-9c7c-5be6a51e0595-kube-rbac-proxy-cm podName:946ce162-a05d-4e23-9c7c-5be6a51e0595 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:31.076383435 +0000 UTC m=+50.599628609 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/946ce162-a05d-4e23-9c7c-5be6a51e0595-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ztp4g" (UID: "946ce162-a05d-4e23-9c7c-5be6a51e0595") : configmap "kube-rbac-proxy" not found Sep 10 16:02:27.076576 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:27.076450 2541 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 16:02:27.076576 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:27.076511 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/946ce162-a05d-4e23-9c7c-5be6a51e0595-insights-runtime-extractor-tls podName:946ce162-a05d-4e23-9c7c-5be6a51e0595 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:31.076494859 +0000 UTC m=+50.599740051 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/946ce162-a05d-4e23-9c7c-5be6a51e0595-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ztp4g" (UID: "946ce162-a05d-4e23-9c7c-5be6a51e0595") : secret "insights-runtime-extractor-tls" not found Sep 10 16:02:27.189897 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:27.189867 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:27.253468 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:27.253433 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:27.429600 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:27.429557 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:27.532976 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:27.532942 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:27.912018 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:27.911970 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:27.976908 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:27.976874 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:29.542478 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:29.542446 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:29.653682 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:29.653648 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:29.756759 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:29.756724 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:31.007289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:31.007259 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:31.112223 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:31.112182 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/946ce162-a05d-4e23-9c7c-5be6a51e0595-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:31.112415 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:31.112288 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/946ce162-a05d-4e23-9c7c-5be6a51e0595-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:31.112415 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:31.112402 2541 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 16:02:31.112556 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:31.112428 2541 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 16:02:31.112556 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:31.112505 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/946ce162-a05d-4e23-9c7c-5be6a51e0595-insights-runtime-extractor-tls podName:946ce162-a05d-4e23-9c7c-5be6a51e0595 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:39.112483103 +0000 UTC m=+58.635728295 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/946ce162-a05d-4e23-9c7c-5be6a51e0595-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ztp4g" (UID: "946ce162-a05d-4e23-9c7c-5be6a51e0595") : secret "insights-runtime-extractor-tls" not found Sep 10 16:02:31.112556 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:31.112531 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/946ce162-a05d-4e23-9c7c-5be6a51e0595-kube-rbac-proxy-cm podName:946ce162-a05d-4e23-9c7c-5be6a51e0595 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:39.112519586 +0000 UTC m=+58.635764764 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/946ce162-a05d-4e23-9c7c-5be6a51e0595-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ztp4g" (UID: "946ce162-a05d-4e23-9c7c-5be6a51e0595") : configmap "kube-rbac-proxy" not found Sep 10 16:02:31.776488 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:31.776450 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:32.216048 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:32.216013 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-bt6g6" Sep 10 16:02:32.216451 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:32.216437 2541 scope.go:117] "RemoveContainer" containerID="bd6d5cea0b409b164a833fc561c49ebf60dec8d4c9059f427fd69a976dab00a8" Sep 10 16:02:32.216595 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:32.216578 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-bt6g6_openshift-console-operator(70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b)\"" pod="openshift-console-operator/console-operator-85b454874b-bt6g6" podUID="70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b" Sep 10 16:02:32.525799 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:32.525678 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:32.525954 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:32.525920 2541 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 16:02:32.525954 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:32.525936 2541 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65b9595f8c-v7mjc: secret "image-registry-tls" not found Sep 10 16:02:32.526034 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:32.526000 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls podName:13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad nodeName:}" failed. No retries permitted until 2026-09-10 16:02:48.525985048 +0000 UTC m=+68.049230222 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls") pod "image-registry-65b9595f8c-v7mjc" (UID: "13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad") : secret "image-registry-tls" not found Sep 10 16:02:32.626872 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:32.626834 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdd6746a-7957-4de4-8241-4c1d740f7814-service-ca-bundle\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:32.627029 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:32.626881 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/befb7be9-0852-4486-a536-219bb03d51f6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-4nbnm\" (UID: \"befb7be9-0852-4486-a536-219bb03d51f6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm" Sep 10 16:02:32.627029 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:32.626912 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39f061d0-48b5-40f0-8428-34d2a6bcc612-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sl7bk\" (UID: \"39f061d0-48b5-40f0-8428-34d2a6bcc612\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sl7bk" Sep 10 16:02:32.627029 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:32.626951 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-metrics-certs\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:32.627029 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:32.627014 2541 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 16:02:32.627156 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:32.627047 2541 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 16:02:32.627156 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:32.627019 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdd6746a-7957-4de4-8241-4c1d740f7814-service-ca-bundle podName:fdd6746a-7957-4de4-8241-4c1d740f7814 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:48.626993985 +0000 UTC m=+68.150239159 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fdd6746a-7957-4de4-8241-4c1d740f7814-service-ca-bundle") pod "router-default-7f48d5849b-h27lv" (UID: "fdd6746a-7957-4de4-8241-4c1d740f7814") : configmap references non-existent config key: service-ca.crt Sep 10 16:02:32.627156 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:32.627078 2541 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 16:02:32.627156 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:32.627140 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f061d0-48b5-40f0-8428-34d2a6bcc612-samples-operator-tls podName:39f061d0-48b5-40f0-8428-34d2a6bcc612 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:48.627126179 +0000 UTC m=+68.150371357 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/39f061d0-48b5-40f0-8428-34d2a6bcc612-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-sl7bk" (UID: "39f061d0-48b5-40f0-8428-34d2a6bcc612") : secret "samples-operator-tls" not found Sep 10 16:02:32.627280 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:32.627158 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/befb7be9-0852-4486-a536-219bb03d51f6-cluster-monitoring-operator-tls podName:befb7be9-0852-4486-a536-219bb03d51f6 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:48.627149153 +0000 UTC m=+68.150394329 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/befb7be9-0852-4486-a536-219bb03d51f6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-4nbnm" (UID: "befb7be9-0852-4486-a536-219bb03d51f6") : secret "cluster-monitoring-operator-tls" not found Sep 10 16:02:32.627280 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:32.627172 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-metrics-certs podName:fdd6746a-7957-4de4-8241-4c1d740f7814 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:48.627164441 +0000 UTC m=+68.150409615 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-metrics-certs") pod "router-default-7f48d5849b-h27lv" (UID: "fdd6746a-7957-4de4-8241-4c1d740f7814") : secret "router-metrics-certs-default" not found Sep 10 16:02:32.728463 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:32.728429 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-metrics-tls\") pod \"dns-default-6r7v6\" (UID: \"0c3d3154-d3b2-4886-9bb6-0036bc9bdd35\") " pod="openshift-dns/dns-default-6r7v6" Sep 10 16:02:32.728622 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:32.728583 2541 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 16:02:32.728965 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:32.728645 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-metrics-tls podName:0c3d3154-d3b2-4886-9bb6-0036bc9bdd35 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:48.728628854 +0000 UTC m=+68.251874029 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-metrics-tls") pod "dns-default-6r7v6" (UID: "0c3d3154-d3b2-4886-9bb6-0036bc9bdd35") : secret "dns-default-metrics-tls" not found Sep 10 16:02:33.933686 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:33.933651 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:34.368787 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:34.368706 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:34.452874 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:34.452840 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:38.016316 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:38.016267 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:38.079512 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:38.079472 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:38.873182 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:38.873148 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:39.177837 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:39.177747 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:39.184923 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:39.184901 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/946ce162-a05d-4e23-9c7c-5be6a51e0595-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:39.185013 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:39.184977 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/946ce162-a05d-4e23-9c7c-5be6a51e0595-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:39.185093 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:39.185076 2541 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 16:02:39.185174 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:39.185162 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/946ce162-a05d-4e23-9c7c-5be6a51e0595-insights-runtime-extractor-tls podName:946ce162-a05d-4e23-9c7c-5be6a51e0595 nodeName:}" failed. No retries permitted until 2026-09-10 16:02:55.185142921 +0000 UTC m=+74.708388109 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/946ce162-a05d-4e23-9c7c-5be6a51e0595-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ztp4g" (UID: "946ce162-a05d-4e23-9c7c-5be6a51e0595") : secret "insights-runtime-extractor-tls" not found Sep 10 16:02:39.185473 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:39.185455 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/946ce162-a05d-4e23-9c7c-5be6a51e0595-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:39.537322 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:39.537228 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:39.621760 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:39.621729 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:39.814150 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:39.814067 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:39.977906 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:39.977877 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:41.179726 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:41.179691 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:41.383071 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:41.383035 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:41.579570 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:41.579492 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:41.772827 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:41.772789 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:41.977730 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:41.977688 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:42.164013 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:42.163986 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r4c6j" Sep 10 16:02:42.586441 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:42.586408 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:42.788814 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:42.788784 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:43.174570 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:43.174536 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:43.241486 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:43.241459 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:43.375828 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:43.375799 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:43.584038 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:43.583950 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:43.653597 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:43.653566 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:43.838456 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:43.838379 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:44.010686 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:44.010644 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:44.134870 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:44.134823 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:44.233475 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:44.233448 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:45.102591 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:45.102558 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:45.202935 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:45.202902 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:45.390871 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:45.390827 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:45.607619 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:45.607582 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:45.794966 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:45.794888 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:45.991335 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:45.991287 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:46.731804 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:46.731771 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:46.749956 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:46.749926 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqbq\" (UniqueName: \"kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq\") pod \"network-check-target-j2m6r\" (UID: \"ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e\") " pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:02:46.752351 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:46.752336 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 16:02:46.762911 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:46.762887 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 16:02:46.774294 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:46.774267 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhqbq\" (UniqueName: \"kubernetes.io/projected/ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e-kube-api-access-rhqbq\") pod \"network-check-target-j2m6r\" (UID: \"ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e\") " pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:02:46.799757 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:46.799733 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g5gt7\"" Sep 10 16:02:46.807503 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:46.807484 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:02:46.851356 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:46.851322 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs\") pod \"network-metrics-daemon-h5frk\" (UID: \"086a5470-5fac-4be6-963c-7b219b25b740\") " pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:02:46.851356 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:46.851364 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret\") pod \"global-pull-secret-syncer-cjj7d\" (UID: \"e3074957-6ddd-4ded-9ddf-7c2fe68929f0\") " pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:02:46.853545 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:46.853522 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 16:02:46.853824 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:46.853807 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 16:02:46.861658 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:46.861587 2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 10 16:02:46.861764 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:02:46.861669 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs podName:086a5470-5fac-4be6-963c-7b219b25b740 nodeName:}" failed. No retries permitted until 2026-09-10 16:03:50.861644966 +0000 UTC m=+130.384890140 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs") pod "network-metrics-daemon-h5frk" (UID: "086a5470-5fac-4be6-963c-7b219b25b740") : secret "metrics-daemon-secret" not found Sep 10 16:02:46.863943 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:46.863923 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3074957-6ddd-4ded-9ddf-7c2fe68929f0-original-pull-secret\") pod \"global-pull-secret-syncer-cjj7d\" (UID: \"e3074957-6ddd-4ded-9ddf-7c2fe68929f0\") " pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:02:46.973620 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:46.973592 2541 scope.go:117] "RemoveContainer" containerID="bd6d5cea0b409b164a833fc561c49ebf60dec8d4c9059f427fd69a976dab00a8" Sep 10 16:02:47.082031 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:47.082002 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j2m6r"] Sep 10 16:02:47.085123 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:02:47.085094 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee16a3cb_e4ad_4f8e_9ea3_6d18923eae5e.slice/crio-14d4e10f281e3382c71fdc9ef00a361a7ccc29f722790a833444b008dab7fede WatchSource:0}: Error finding container 14d4e10f281e3382c71fdc9ef00a361a7ccc29f722790a833444b008dab7fede: Status 404 returned error can't find the container with id 14d4e10f281e3382c71fdc9ef00a361a7ccc29f722790a833444b008dab7fede Sep 10 16:02:47.104715 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:47.104686 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjj7d" Sep 10 16:02:47.195089 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:47.195051 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:47.293083 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:47.293006 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j2m6r" event={"ID":"ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e","Type":"ContainerStarted","Data":"14d4e10f281e3382c71fdc9ef00a361a7ccc29f722790a833444b008dab7fede"} Sep 10 16:02:47.294948 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:47.294925 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/1.log" Sep 10 16:02:47.295044 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:47.295003 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-bt6g6" event={"ID":"70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b","Type":"ContainerStarted","Data":"ddadcd4deddc366fd2fcf8eadd2e1b6bf22f9a225980e7ef3aa924486aa3f5d3"} Sep 10 16:02:47.295280 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:47.295265 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-bt6g6" Sep 10 16:02:47.374932 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:47.374898 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cjj7d"] Sep 10 16:02:47.378905 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:02:47.378874 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3074957_6ddd_4ded_9ddf_7c2fe68929f0.slice/crio-c7f535222b42133f68fc19eea02f2c10e065c408dccc6213e5518ffd0a83fc14 WatchSource:0}: Error finding container c7f535222b42133f68fc19eea02f2c10e065c408dccc6213e5518ffd0a83fc14: Status 404 returned error can't find the container with id c7f535222b42133f68fc19eea02f2c10e065c408dccc6213e5518ffd0a83fc14 Sep 10 16:02:47.578969 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:47.578883 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:47.759127 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:47.759096 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-bt6g6" Sep 10 16:02:48.186526 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.186492 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:48.285780 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.285737 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:48.301134 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.301093 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cjj7d" event={"ID":"e3074957-6ddd-4ded-9ddf-7c2fe68929f0","Type":"ContainerStarted","Data":"c7f535222b42133f68fc19eea02f2c10e065c408dccc6213e5518ffd0a83fc14"} Sep 10 16:02:48.568021 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.567931 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:48.571336 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.571293 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls\") pod \"image-registry-65b9595f8c-v7mjc\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:48.595933 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.595898 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:48.668479 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.668441 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdd6746a-7957-4de4-8241-4c1d740f7814-service-ca-bundle\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:48.668724 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.668492 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/befb7be9-0852-4486-a536-219bb03d51f6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-4nbnm\" (UID: \"befb7be9-0852-4486-a536-219bb03d51f6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm" Sep 10 16:02:48.668724 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.668518 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39f061d0-48b5-40f0-8428-34d2a6bcc612-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sl7bk\" (UID: \"39f061d0-48b5-40f0-8428-34d2a6bcc612\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sl7bk" Sep 10 16:02:48.668724 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.668551 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-metrics-certs\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:48.669256 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.669209 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdd6746a-7957-4de4-8241-4c1d740f7814-service-ca-bundle\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:48.671573 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.671545 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/befb7be9-0852-4486-a536-219bb03d51f6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-4nbnm\" (UID: \"befb7be9-0852-4486-a536-219bb03d51f6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm" Sep 10 16:02:48.671700 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.671633 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39f061d0-48b5-40f0-8428-34d2a6bcc612-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sl7bk\" (UID: \"39f061d0-48b5-40f0-8428-34d2a6bcc612\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sl7bk" Sep 10 16:02:48.671759 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.671697 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd6746a-7957-4de4-8241-4c1d740f7814-metrics-certs\") pod \"router-default-7f48d5849b-h27lv\" (UID: \"fdd6746a-7957-4de4-8241-4c1d740f7814\") " pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:48.769366 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.769293 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-metrics-tls\") pod \"dns-default-6r7v6\" (UID: \"0c3d3154-d3b2-4886-9bb6-0036bc9bdd35\") " pod="openshift-dns/dns-default-6r7v6" Sep 10 16:02:48.772450 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.772419 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c3d3154-d3b2-4886-9bb6-0036bc9bdd35-metrics-tls\") pod \"dns-default-6r7v6\" (UID: \"0c3d3154-d3b2-4886-9bb6-0036bc9bdd35\") " pod="openshift-dns/dns-default-6r7v6" Sep 10 16:02:48.822171 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.822081 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nqhnk\"" Sep 10 16:02:48.830066 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.830029 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:48.862470 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.862442 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-llg5z\"" Sep 10 16:02:48.870544 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.870515 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm" Sep 10 16:02:48.882035 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.882006 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gm4g7\"" Sep 10 16:02:48.890243 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.890214 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sl7bk" Sep 10 16:02:48.906347 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.905596 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-l5xqb\"" Sep 10 16:02:48.907910 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.907760 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:48.917046 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.916594 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:48.966008 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.965767 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fqpr5\"" Sep 10 16:02:48.968686 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:48.968232 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6r7v6" Sep 10 16:02:49.008285 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.008128 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:49.104380 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.104225 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:49.146981 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.146890 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65b9595f8c-v7mjc"] Sep 10 16:02:49.152728 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:02:49.152701 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f79cf2_a9e2_45ac_8f66_9a5e35cb7fad.slice/crio-51f5b5bd835ccec1f7b7fe0b6a23c4dc2595bd57e4a818a3faa0c60cf0a02b96 WatchSource:0}: Error finding container 51f5b5bd835ccec1f7b7fe0b6a23c4dc2595bd57e4a818a3faa0c60cf0a02b96: Status 404 returned error can't find the container with id 51f5b5bd835ccec1f7b7fe0b6a23c4dc2595bd57e4a818a3faa0c60cf0a02b96 Sep 10 16:02:49.171778 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.171742 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm"] Sep 10 16:02:49.175778 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:02:49.175750 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbefb7be9_0852_4486_a536_219bb03d51f6.slice/crio-78793ba17175435e92f32a467d0b504115522e7648b8e71a566f695c57c92f79 WatchSource:0}: Error finding container 78793ba17175435e92f32a467d0b504115522e7648b8e71a566f695c57c92f79: Status 404 returned error can't find the container with id 78793ba17175435e92f32a467d0b504115522e7648b8e71a566f695c57c92f79 Sep 10 16:02:49.187810 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.187772 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:49.222238 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.222203 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f48d5849b-h27lv"] Sep 10 16:02:49.224627 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.224604 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sl7bk"] Sep 10 16:02:49.227706 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:02:49.227663 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd6746a_7957_4de4_8241_4c1d740f7814.slice/crio-e102be89004edcfad2704b4f7a9b9457c72e930ab69ef307e71c6b45b027a17c WatchSource:0}: Error finding container e102be89004edcfad2704b4f7a9b9457c72e930ab69ef307e71c6b45b027a17c: Status 404 returned error can't find the container with id e102be89004edcfad2704b4f7a9b9457c72e930ab69ef307e71c6b45b027a17c Sep 10 16:02:49.281044 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.281014 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6r7v6"] Sep 10 16:02:49.285903 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:02:49.285874 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c3d3154_d3b2_4886_9bb6_0036bc9bdd35.slice/crio-bae1c634b30cda0217232d3112cf8d47e4e253b7c40176dd735544259a5d8763 WatchSource:0}: Error finding container bae1c634b30cda0217232d3112cf8d47e4e253b7c40176dd735544259a5d8763: Status 404 returned error can't find the container with id bae1c634b30cda0217232d3112cf8d47e4e253b7c40176dd735544259a5d8763 Sep 10 16:02:49.311176 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.311086 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm" event={"ID":"befb7be9-0852-4486-a536-219bb03d51f6","Type":"ContainerStarted","Data":"78793ba17175435e92f32a467d0b504115522e7648b8e71a566f695c57c92f79"} Sep 10 16:02:49.314119 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.314027 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sl7bk" event={"ID":"39f061d0-48b5-40f0-8428-34d2a6bcc612","Type":"ContainerStarted","Data":"623fd140cd14d587e20ebb5bfc7e6ec02f6a926f03f5605034bc15ba7862866b"} Sep 10 16:02:49.315683 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.315604 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6r7v6" event={"ID":"0c3d3154-d3b2-4886-9bb6-0036bc9bdd35","Type":"ContainerStarted","Data":"bae1c634b30cda0217232d3112cf8d47e4e253b7c40176dd735544259a5d8763"} Sep 10 16:02:49.318395 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.317929 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" event={"ID":"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad","Type":"ContainerStarted","Data":"7d409c6a977115acc3c1f72eea4420b65e2b5733b5f096882a076d79614da17d"} Sep 10 16:02:49.318395 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.317973 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" event={"ID":"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad","Type":"ContainerStarted","Data":"51f5b5bd835ccec1f7b7fe0b6a23c4dc2595bd57e4a818a3faa0c60cf0a02b96"} Sep 10 16:02:49.318395 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.318213 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:02:49.320435 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.320354 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f48d5849b-h27lv" event={"ID":"fdd6746a-7957-4de4-8241-4c1d740f7814","Type":"ContainerStarted","Data":"e102be89004edcfad2704b4f7a9b9457c72e930ab69ef307e71c6b45b027a17c"} Sep 10 16:02:49.345216 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.345149 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" podStartSLOduration=67.345126971 podStartE2EDuration="1m7.345126971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:49.344658259 +0000 UTC m=+68.867903456" watchObservedRunningTime="2026-09-10 16:02:49.345126971 +0000 UTC m=+68.868372170" Sep 10 16:02:49.371606 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.370823 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7f48d5849b-h27lv" podStartSLOduration=34.370802977 podStartE2EDuration="34.370802977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:49.370038321 +0000 UTC m=+68.893283715" watchObservedRunningTime="2026-09-10 16:02:49.370802977 +0000 UTC m=+68.894048172" Sep 10 16:02:49.377837 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.377768 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:49.464166 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.464088 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:49.558769 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.558732 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:49.662404 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.662291 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:49.728615 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.728585 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:49.844892 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.844860 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:49.917612 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.917521 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:49.920771 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.920742 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:49.987095 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:49.987061 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:50.063023 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:50.062981 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:50.325396 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:50.325280 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f48d5849b-h27lv" event={"ID":"fdd6746a-7957-4de4-8241-4c1d740f7814","Type":"ContainerStarted","Data":"c68ffd1e93c018095f15250fd82d4d4f41189a5af59adbd97f9492371f7236b4"} Sep 10 16:02:50.325646 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:50.325603 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:50.326964 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:50.326942 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7f48d5849b-h27lv" Sep 10 16:02:51.459705 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:51.459671 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:51.554168 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:51.554131 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:53.031848 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.031771 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:53.129578 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.129546 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:53.341761 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.341650 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j2m6r" event={"ID":"ee16a3cb-e4ad-4f8e-9ea3-6d18923eae5e","Type":"ContainerStarted","Data":"923077f7c6c77229da268b74eb93f11622179bfce71f1f1caf2e394adf7474da"} Sep 10 16:02:53.341937 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.341844 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:02:53.344236 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.343814 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cjj7d" event={"ID":"e3074957-6ddd-4ded-9ddf-7c2fe68929f0","Type":"ContainerStarted","Data":"b9b5ce6ee1d28fe7d06d33570079ab9f4d1a3b72ed88792c43d1650e9e9a8605"} Sep 10 16:02:53.346323 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.346273 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sl7bk" event={"ID":"39f061d0-48b5-40f0-8428-34d2a6bcc612","Type":"ContainerStarted","Data":"3476bbec7344450f6ddc25684567f88927d2657e3d7f0aa06d8095cd0c6ce854"} Sep 10 16:02:53.346463 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.346345 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sl7bk" event={"ID":"39f061d0-48b5-40f0-8428-34d2a6bcc612","Type":"ContainerStarted","Data":"14b40e2b565afb3405db252e92bf23f2e1044b4d52008e7f70d0305d90d9bc4a"} Sep 10 16:02:53.422505 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.421838 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j2m6r" podStartSLOduration=66.865128948 podStartE2EDuration="1m12.421814792s" totalImagesPullingTime="5.556685844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:53.376433804 +0000 UTC m=+72.899679014" watchObservedRunningTime="2026-09-10 16:02:53.421814792 +0000 UTC m=+72.945059988" Sep 10 16:02:53.422505 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.422389 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-c4p62"] Sep 10 16:02:53.422856 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.422823 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cjj7d" podStartSLOduration=66.117116598 podStartE2EDuration="1m11.422813029s" totalImagesPullingTime="5.305696431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:53.421039162 +0000 UTC m=+72.944284359" watchObservedRunningTime="2026-09-10 16:02:53.422813029 +0000 UTC m=+72.946058226" Sep 10 16:02:53.427342 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.427292 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-c4p62" Sep 10 16:02:53.430235 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.430203 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 10 16:02:53.430985 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.430834 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 10 16:02:53.430985 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.430875 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-vcbh7\"" Sep 10 16:02:53.447146 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.447066 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-c4p62"] Sep 10 16:02:53.460032 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.459978 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sl7bk" podStartSLOduration=35.043348744 podStartE2EDuration="38.459960984s" totalImagesPullingTime="3.41661224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:02:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:53.459555366 +0000 UTC m=+72.982800773" watchObservedRunningTime="2026-09-10 16:02:53.459960984 +0000 UTC m=+72.983206182" Sep 10 16:02:53.514634 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.514603 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/30ac85d4-8250-4700-9d2b-c19e029b3ddc-nginx-conf\") pod \"networking-console-plugin-79977f9844-c4p62\" (UID: \"30ac85d4-8250-4700-9d2b-c19e029b3ddc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-c4p62" Sep 10 16:02:53.514634 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.514639 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30ac85d4-8250-4700-9d2b-c19e029b3ddc-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-c4p62\" (UID: \"30ac85d4-8250-4700-9d2b-c19e029b3ddc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-c4p62" Sep 10 16:02:53.523560 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.523526 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-vmph9"] Sep 10 16:02:53.528069 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.528046 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-vmph9" Sep 10 16:02:53.530491 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.530467 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 16:02:53.530601 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.530467 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-hhcqs\"" Sep 10 16:02:53.533159 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.531096 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 16:02:53.544177 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.544151 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-vmph9"] Sep 10 16:02:53.615285 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.615190 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/30ac85d4-8250-4700-9d2b-c19e029b3ddc-nginx-conf\") pod \"networking-console-plugin-79977f9844-c4p62\" (UID: \"30ac85d4-8250-4700-9d2b-c19e029b3ddc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-c4p62" Sep 10 16:02:53.615285 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.615239 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30ac85d4-8250-4700-9d2b-c19e029b3ddc-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-c4p62\" (UID: \"30ac85d4-8250-4700-9d2b-c19e029b3ddc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-c4p62" Sep 10 16:02:53.615285 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.615274 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwfdr\" (UniqueName: \"kubernetes.io/projected/03fe5d54-88f0-49bf-ba0b-96e56def25dc-kube-api-access-kwfdr\") pod \"downloads-85d98c998f-vmph9\" (UID: \"03fe5d54-88f0-49bf-ba0b-96e56def25dc\") " pod="openshift-console/downloads-85d98c998f-vmph9" Sep 10 16:02:53.615970 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.615943 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/30ac85d4-8250-4700-9d2b-c19e029b3ddc-nginx-conf\") pod \"networking-console-plugin-79977f9844-c4p62\" (UID: \"30ac85d4-8250-4700-9d2b-c19e029b3ddc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-c4p62" Sep 10 16:02:53.618243 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.618225 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30ac85d4-8250-4700-9d2b-c19e029b3ddc-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-c4p62\" (UID: \"30ac85d4-8250-4700-9d2b-c19e029b3ddc\") " pod="openshift-network-console/networking-console-plugin-79977f9844-c4p62" Sep 10 16:02:53.716568 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.716530 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwfdr\" (UniqueName: \"kubernetes.io/projected/03fe5d54-88f0-49bf-ba0b-96e56def25dc-kube-api-access-kwfdr\") pod \"downloads-85d98c998f-vmph9\" (UID: \"03fe5d54-88f0-49bf-ba0b-96e56def25dc\") " pod="openshift-console/downloads-85d98c998f-vmph9" Sep 10 16:02:53.738106 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.738069 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-c4p62" Sep 10 16:02:53.739550 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.739530 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwfdr\" (UniqueName: \"kubernetes.io/projected/03fe5d54-88f0-49bf-ba0b-96e56def25dc-kube-api-access-kwfdr\") pod \"downloads-85d98c998f-vmph9\" (UID: \"03fe5d54-88f0-49bf-ba0b-96e56def25dc\") " pod="openshift-console/downloads-85d98c998f-vmph9" Sep 10 16:02:53.813865 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.813825 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:53.841060 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.841021 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-vmph9" Sep 10 16:02:53.959533 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:53.959497 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:54.244926 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:54.244827 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-c4p62"] Sep 10 16:02:54.246023 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:54.246002 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-vmph9"] Sep 10 16:02:54.248044 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:02:54.248022 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03fe5d54_88f0_49bf_ba0b_96e56def25dc.slice/crio-a9ee65948eb53f779d37fb7db3d39d48c0f57749d5fd04353cc9b7423f917aee WatchSource:0}: Error finding container a9ee65948eb53f779d37fb7db3d39d48c0f57749d5fd04353cc9b7423f917aee: Status 404 returned error can't find the container with id a9ee65948eb53f779d37fb7db3d39d48c0f57749d5fd04353cc9b7423f917aee Sep 10 16:02:54.249652 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:02:54.249626 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ac85d4_8250_4700_9d2b_c19e029b3ddc.slice/crio-b11fe2137aeff9cd48dfbe20f48ea113902be40e97212b042267d5ad8281b2bd WatchSource:0}: Error finding container b11fe2137aeff9cd48dfbe20f48ea113902be40e97212b042267d5ad8281b2bd: Status 404 returned error can't find the container with id b11fe2137aeff9cd48dfbe20f48ea113902be40e97212b042267d5ad8281b2bd Sep 10 16:02:54.351183 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:54.351145 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-c4p62" event={"ID":"30ac85d4-8250-4700-9d2b-c19e029b3ddc","Type":"ContainerStarted","Data":"b11fe2137aeff9cd48dfbe20f48ea113902be40e97212b042267d5ad8281b2bd"} Sep 10 16:02:54.352824 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:54.352799 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm" event={"ID":"befb7be9-0852-4486-a536-219bb03d51f6","Type":"ContainerStarted","Data":"c03983a89448c9c501d6a922b5855193165bc306067f3b0ca2f1d504ffe4006e"} Sep 10 16:02:54.354065 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:54.354043 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-vmph9" event={"ID":"03fe5d54-88f0-49bf-ba0b-96e56def25dc","Type":"ContainerStarted","Data":"a9ee65948eb53f779d37fb7db3d39d48c0f57749d5fd04353cc9b7423f917aee"} Sep 10 16:02:54.356005 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:54.355977 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6r7v6" event={"ID":"0c3d3154-d3b2-4886-9bb6-0036bc9bdd35","Type":"ContainerStarted","Data":"980a840157230b247657e26bbc8bd0cc65bf6afe7f958b1d55f1ab6a9a2e7e6e"} Sep 10 16:02:54.356115 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:54.356020 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6r7v6" event={"ID":"0c3d3154-d3b2-4886-9bb6-0036bc9bdd35","Type":"ContainerStarted","Data":"38cceea10f1a2c4a636fc77e3cf9e15a5abe6c57bb8d2fc2e42e2bcdacbcd3ec"} Sep 10 16:02:54.356115 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:54.356041 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6r7v6" Sep 10 16:02:54.377591 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:54.377546 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-4nbnm" podStartSLOduration=38.600508001 podStartE2EDuration="43.377531353s" totalImagesPullingTime="4.777023352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:54.377039412 +0000 UTC m=+73.900284633" watchObservedRunningTime="2026-09-10 16:02:54.377531353 +0000 UTC m=+73.900776552" Sep 10 16:02:54.405433 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:54.405255 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6r7v6" podStartSLOduration=33.741427754 podStartE2EDuration="38.405230354s" totalImagesPullingTime="4.6638026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:02:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:54.404573353 +0000 UTC m=+73.927818544" watchObservedRunningTime="2026-09-10 16:02:54.405230354 +0000 UTC m=+73.928475554" Sep 10 16:02:54.448548 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:54.448512 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:54.545648 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:54.545575 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:54.783083 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:54.782995 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:54.902654 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:54.902615 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:55.005818 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:55.005779 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:55.232123 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:55.232025 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/946ce162-a05d-4e23-9c7c-5be6a51e0595-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:55.241742 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:55.240054 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/946ce162-a05d-4e23-9c7c-5be6a51e0595-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ztp4g\" (UID: \"946ce162-a05d-4e23-9c7c-5be6a51e0595\") " pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:55.400756 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:55.400724 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6s77g\"" Sep 10 16:02:55.409389 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:55.409351 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ztp4g" Sep 10 16:02:55.835583 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:55.835548 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:55.869185 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:55.869112 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6459747688-mgvzg"] Sep 10 16:02:55.889312 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:55.889272 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6459747688-mgvzg"] Sep 10 16:02:55.889463 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:55.889406 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:02:55.891659 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:55.891635 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 16:02:55.891659 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:55.891653 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 16:02:55.891850 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:55.891674 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 16:02:55.892216 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:55.892194 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 16:02:55.892474 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:55.892459 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 16:02:55.892652 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:55.892638 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2k588\"" Sep 10 16:02:55.895023 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:55.895001 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ztp4g"] Sep 10 16:02:55.944262 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:02:55.944228 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946ce162_a05d_4e23_9c7c_5be6a51e0595.slice/crio-a59391a0241d7f343e6d8d37b4da22514c958fa6b2bef6f53b3cdd419b5673eb WatchSource:0}: Error finding container a59391a0241d7f343e6d8d37b4da22514c958fa6b2bef6f53b3cdd419b5673eb: Status 404 returned error can't find the container with id a59391a0241d7f343e6d8d37b4da22514c958fa6b2bef6f53b3cdd419b5673eb Sep 10 16:02:55.945362 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:55.945328 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:56.039313 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.039277 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-console-serving-cert\") pod \"console-6459747688-mgvzg\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:02:56.039432 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.039359 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-oauth-serving-cert\") pod \"console-6459747688-mgvzg\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:02:56.039432 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.039414 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-console-config\") pod \"console-6459747688-mgvzg\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:02:56.039545 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.039478 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb2vx\" (UniqueName: \"kubernetes.io/projected/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-kube-api-access-rb2vx\") pod \"console-6459747688-mgvzg\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:02:56.039545 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.039500 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-service-ca\") pod \"console-6459747688-mgvzg\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:02:56.039545 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.039522 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-console-oauth-config\") pod \"console-6459747688-mgvzg\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:02:56.140350 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.140290 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-oauth-serving-cert\") pod \"console-6459747688-mgvzg\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:02:56.140522 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.140384 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-console-config\") pod \"console-6459747688-mgvzg\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:02:56.140522 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.140436 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb2vx\" (UniqueName: \"kubernetes.io/projected/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-kube-api-access-rb2vx\") pod \"console-6459747688-mgvzg\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:02:56.140522 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.140469 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-service-ca\") pod \"console-6459747688-mgvzg\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:02:56.140678 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.140524 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-console-oauth-config\") pod \"console-6459747688-mgvzg\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:02:56.140678 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.140571 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-console-serving-cert\") pod \"console-6459747688-mgvzg\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:02:56.141052 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.141021 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-oauth-serving-cert\") pod \"console-6459747688-mgvzg\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:02:56.141207 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.141169 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-console-config\") pod \"console-6459747688-mgvzg\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:02:56.141810 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.141767 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-service-ca\") pod \"console-6459747688-mgvzg\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:02:56.143345 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.143325 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-console-oauth-config\") pod \"console-6459747688-mgvzg\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:02:56.143674 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.143646 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-console-serving-cert\") pod \"console-6459747688-mgvzg\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:02:56.155006 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.154983 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb2vx\" (UniqueName: \"kubernetes.io/projected/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-kube-api-access-rb2vx\") pod \"console-6459747688-mgvzg\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:02:56.201948 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.201907 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:02:56.331564 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.331508 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:56.367611 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.367561 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-c4p62" event={"ID":"30ac85d4-8250-4700-9d2b-c19e029b3ddc","Type":"ContainerStarted","Data":"a241dc7694a49887bb748baf074f185826052b2d90d7ab72cd7460f5b7044844"} Sep 10 16:02:56.369971 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.369933 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ztp4g" event={"ID":"946ce162-a05d-4e23-9c7c-5be6a51e0595","Type":"ContainerStarted","Data":"f5d6a937480916b35a90da59d7a1333946048e1bdcee8853c08758d742c2d46a"} Sep 10 16:02:56.370104 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.369989 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ztp4g" event={"ID":"946ce162-a05d-4e23-9c7c-5be6a51e0595","Type":"ContainerStarted","Data":"a59391a0241d7f343e6d8d37b4da22514c958fa6b2bef6f53b3cdd419b5673eb"} Sep 10 16:02:56.399514 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.399403 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-c4p62" podStartSLOduration=1.677232198 podStartE2EDuration="3.399382198s" totalImagesPullingTime="1.72215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:56.398654212 +0000 UTC m=+75.921899409" watchObservedRunningTime="2026-09-10 16:02:56.399382198 +0000 UTC m=+75.922627394" Sep 10 16:02:56.413894 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.413838 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:56.511883 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:56.511843 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6459747688-mgvzg"] Sep 10 16:02:56.515537 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:02:56.515511 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59656fa7_1ec7_40d1_bbbc_73927c3f4ff6.slice/crio-3a330319508d5db0337f96c623f37f27d743118c70d8f03a7172bc1e17158d57 WatchSource:0}: Error finding container 3a330319508d5db0337f96c623f37f27d743118c70d8f03a7172bc1e17158d57: Status 404 returned error can't find the container with id 3a330319508d5db0337f96c623f37f27d743118c70d8f03a7172bc1e17158d57 Sep 10 16:02:57.301957 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:57.301873 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:57.377846 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:57.377776 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6459747688-mgvzg" event={"ID":"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6","Type":"ContainerStarted","Data":"3a330319508d5db0337f96c623f37f27d743118c70d8f03a7172bc1e17158d57"} Sep 10 16:02:57.381374 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:57.381269 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ztp4g" event={"ID":"946ce162-a05d-4e23-9c7c-5be6a51e0595","Type":"ContainerStarted","Data":"5e5b29192e17f41368c7e8ade87cf03dbf6ffe0ce5c3ed87f1647b4dc4cec5d8"} Sep 10 16:02:57.560900 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:57.560810 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:57.659632 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:57.659601 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:59.809140 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:59.809056 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:59.878157 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:59.878114 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-t2txz"] Sep 10 16:02:59.887030 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:59.886994 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-t2txz" Sep 10 16:02:59.889567 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:59.889537 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 10 16:02:59.889728 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:59.889620 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gh7sq\"" Sep 10 16:02:59.890072 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:59.890046 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 16:02:59.890191 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:59.890171 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 10 16:02:59.903573 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:59.903507 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-t2txz"] Sep 10 16:02:59.915107 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:59.915049 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:59.978970 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:59.978932 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mss8s\" (UniqueName: \"kubernetes.io/projected/8f659978-e772-408c-9ecf-59e161eab712-kube-api-access-mss8s\") pod \"prometheus-operator-6f4946db56-t2txz\" (UID: \"8f659978-e772-408c-9ecf-59e161eab712\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t2txz" Sep 10 16:02:59.979156 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:59.979009 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f659978-e772-408c-9ecf-59e161eab712-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-t2txz\" (UID: \"8f659978-e772-408c-9ecf-59e161eab712\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t2txz" Sep 10 16:02:59.979156 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:59.979072 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f659978-e772-408c-9ecf-59e161eab712-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-t2txz\" (UID: \"8f659978-e772-408c-9ecf-59e161eab712\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t2txz" Sep 10 16:02:59.979156 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:02:59.979120 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f659978-e772-408c-9ecf-59e161eab712-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-t2txz\" (UID: \"8f659978-e772-408c-9ecf-59e161eab712\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t2txz" Sep 10 16:03:00.079549 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:00.079520 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f659978-e772-408c-9ecf-59e161eab712-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-t2txz\" (UID: \"8f659978-e772-408c-9ecf-59e161eab712\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t2txz" Sep 10 16:03:00.079679 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:00.079587 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f659978-e772-408c-9ecf-59e161eab712-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-t2txz\" (UID: \"8f659978-e772-408c-9ecf-59e161eab712\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t2txz" Sep 10 16:03:00.079679 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:00.079638 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f659978-e772-408c-9ecf-59e161eab712-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-t2txz\" (UID: \"8f659978-e772-408c-9ecf-59e161eab712\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t2txz" Sep 10 16:03:00.079679 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:00.079671 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mss8s\" (UniqueName: \"kubernetes.io/projected/8f659978-e772-408c-9ecf-59e161eab712-kube-api-access-mss8s\") pod \"prometheus-operator-6f4946db56-t2txz\" (UID: \"8f659978-e772-408c-9ecf-59e161eab712\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t2txz" Sep 10 16:03:00.080725 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:00.080694 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f659978-e772-408c-9ecf-59e161eab712-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-t2txz\" (UID: \"8f659978-e772-408c-9ecf-59e161eab712\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t2txz" Sep 10 16:03:00.082882 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:00.082838 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f659978-e772-408c-9ecf-59e161eab712-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-t2txz\" (UID: \"8f659978-e772-408c-9ecf-59e161eab712\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t2txz" Sep 10 16:03:00.082882 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:00.082848 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f659978-e772-408c-9ecf-59e161eab712-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-t2txz\" (UID: \"8f659978-e772-408c-9ecf-59e161eab712\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t2txz" Sep 10 16:03:00.094831 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:00.094807 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mss8s\" (UniqueName: \"kubernetes.io/projected/8f659978-e772-408c-9ecf-59e161eab712-kube-api-access-mss8s\") pod \"prometheus-operator-6f4946db56-t2txz\" (UID: \"8f659978-e772-408c-9ecf-59e161eab712\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t2txz" Sep 10 16:03:00.132639 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:00.132607 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:00.202457 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:00.202417 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-t2txz" Sep 10 16:03:00.403260 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:00.403214 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ztp4g" event={"ID":"946ce162-a05d-4e23-9c7c-5be6a51e0595","Type":"ContainerStarted","Data":"5c8ed2f067501e23eb2fd39b8d14a139432a659b972559da3705f612316918a1"} Sep 10 16:03:00.405535 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:00.405500 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6459747688-mgvzg" event={"ID":"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6","Type":"ContainerStarted","Data":"bee3170929383d671753ca8ec599354a4258eda3b810b4f5cbf513afef60b6d4"} Sep 10 16:03:00.425322 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:00.425228 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ztp4g" podStartSLOduration=33.556767338 podStartE2EDuration="37.425212787s" totalImagesPullingTime="3.868445449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:02:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:03:00.424950506 +0000 UTC m=+79.948195716" watchObservedRunningTime="2026-09-10 16:03:00.425212787 +0000 UTC m=+79.948457983" Sep 10 16:03:00.451054 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:00.450998 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6459747688-mgvzg" podStartSLOduration=1.911877802 podStartE2EDuration="5.450978121s" totalImagesPullingTime="3.539100319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:03:00.449419511 +0000 UTC m=+79.972664711" watchObservedRunningTime="2026-09-10 16:03:00.450978121 +0000 UTC m=+79.974223319" Sep 10 16:03:00.487049 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:00.487014 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-t2txz"] Sep 10 16:03:00.490605 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:03:00.490567 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f659978_e772_408c_9ecf_59e161eab712.slice/crio-0e30853205e67cc4395e43fcf1494b2f8c8f1e8addacaa986d43c1258bbf3131 WatchSource:0}: Error finding container 0e30853205e67cc4395e43fcf1494b2f8c8f1e8addacaa986d43c1258bbf3131: Status 404 returned error can't find the container with id 0e30853205e67cc4395e43fcf1494b2f8c8f1e8addacaa986d43c1258bbf3131 Sep 10 16:03:01.412184 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:01.412141 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-t2txz" event={"ID":"8f659978-e772-408c-9ecf-59e161eab712","Type":"ContainerStarted","Data":"0e30853205e67cc4395e43fcf1494b2f8c8f1e8addacaa986d43c1258bbf3131"} Sep 10 16:03:02.108864 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:02.108777 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:02.418890 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:02.418841 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-t2txz" event={"ID":"8f659978-e772-408c-9ecf-59e161eab712","Type":"ContainerStarted","Data":"1af3e355432746ecf901ceb04ae83063a9d69c32cfdc49c1dd1778efbcc00f88"} Sep 10 16:03:02.419397 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:02.418910 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-t2txz" event={"ID":"8f659978-e772-408c-9ecf-59e161eab712","Type":"ContainerStarted","Data":"fffdb35bf1e1a940712ae0fa315ac59490ac047e961703c9ee2c223a61b73b38"} Sep 10 16:03:02.444287 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:02.444210 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-t2txz" podStartSLOduration=2.235307784 podStartE2EDuration="3.444187154s" totalImagesPullingTime="1.20887937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:03:02.442799746 +0000 UTC m=+81.966044945" watchObservedRunningTime="2026-09-10 16:03:02.444187154 +0000 UTC m=+81.967432351" Sep 10 16:03:02.546640 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:02.546580 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:02.647676 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:02.647635 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:04.142683 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:04.142628 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:04.254698 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:04.254628 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:04.362291 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:04.362259 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6r7v6" Sep 10 16:03:04.789509 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:04.789471 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:04.961827 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:04.961797 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:04.985561 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:04.985526 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vblpz"] Sep 10 16:03:04.990339 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:04.990314 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:04.993665 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:04.993640 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-75jhs\"" Sep 10 16:03:04.993802 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:04.993688 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 16:03:04.993802 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:04.993701 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 16:03:04.994172 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:04.994150 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 16:03:05.075398 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.075294 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:05.128581 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.128541 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-metrics-client-ca\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.128749 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.128609 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-sys\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.128749 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.128639 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.128749 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.128663 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q7q6\" (UniqueName: \"kubernetes.io/projected/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-kube-api-access-4q7q6\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.128749 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.128698 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-node-exporter-textfile\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.128749 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.128727 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-node-exporter-accelerators-collector-config\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.128975 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.128756 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-node-exporter-wtmp\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.128975 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.128848 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-node-exporter-tls\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.128975 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.128914 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-root\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.229440 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.229397 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-node-exporter-accelerators-collector-config\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.229440 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.229451 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-node-exporter-wtmp\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.230110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.229496 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-node-exporter-tls\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.230110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.229530 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-root\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.230110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.229590 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-metrics-client-ca\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.230110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.229624 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-sys\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.230110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.229652 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.230110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.229675 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q7q6\" (UniqueName: \"kubernetes.io/projected/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-kube-api-access-4q7q6\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.230110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.229670 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-node-exporter-wtmp\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.230110 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:03:05.229677 2541 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 10 16:03:05.230110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.229720 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-sys\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.230110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.229731 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-node-exporter-textfile\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.230110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.229685 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-root\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.230110 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:03:05.229764 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-node-exporter-tls podName:0ab59368-68c9-4b6b-a61f-6cf978e2fb30 nodeName:}" failed. No retries permitted until 2026-09-10 16:03:05.729748626 +0000 UTC m=+85.252993801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-node-exporter-tls") pod "node-exporter-vblpz" (UID: "0ab59368-68c9-4b6b-a61f-6cf978e2fb30") : secret "node-exporter-tls" not found Sep 10 16:03:05.230110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.230026 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-node-exporter-textfile\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.230110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.230093 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-node-exporter-accelerators-collector-config\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.230681 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.230154 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-metrics-client-ca\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.232564 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.232544 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.247849 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.247823 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q7q6\" (UniqueName: \"kubernetes.io/projected/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-kube-api-access-4q7q6\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.426647 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.426607 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:05.633604 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.633566 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:05.733678 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.733582 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-node-exporter-tls\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.736358 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.736336 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ab59368-68c9-4b6b-a61f-6cf978e2fb30-node-exporter-tls\") pod \"node-exporter-vblpz\" (UID: \"0ab59368-68c9-4b6b-a61f-6cf978e2fb30\") " pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:05.858608 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.858573 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 16:03:05.863930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.863905 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:05.866061 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.866035 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 16:03:05.866061 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.866060 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 16:03:05.866219 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.866076 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 16:03:05.866634 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.866410 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 16:03:05.866634 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.866412 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 16:03:05.866634 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.866513 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 16:03:05.866854 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.866702 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 16:03:05.866854 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.866742 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 16:03:05.866854 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.866750 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6p9l6\"" Sep 10 16:03:05.868769 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.868752 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 16:03:05.885025 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.885002 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 16:03:05.901534 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:05.901509 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vblpz" Sep 10 16:03:06.036149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.036061 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.036149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.036108 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.036149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.036135 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1e2088c6-e4c2-485a-b3c7-094b1e42b402-config-out\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.036441 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.036218 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.036441 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.036257 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.036441 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.036339 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1e2088c6-e4c2-485a-b3c7-094b1e42b402-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.036441 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.036361 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e2088c6-e4c2-485a-b3c7-094b1e42b402-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.036441 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.036392 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.036656 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.036452 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-config-volume\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.036656 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.036504 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1e2088c6-e4c2-485a-b3c7-094b1e42b402-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.036656 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.036531 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt5xq\" (UniqueName: \"kubernetes.io/projected/1e2088c6-e4c2-485a-b3c7-094b1e42b402-kube-api-access-tt5xq\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.036656 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.036556 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-web-config\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.036656 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.036585 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e2088c6-e4c2-485a-b3c7-094b1e42b402-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.137481 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.137437 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-config-volume\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.137664 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.137505 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1e2088c6-e4c2-485a-b3c7-094b1e42b402-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.137664 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.137540 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt5xq\" (UniqueName: \"kubernetes.io/projected/1e2088c6-e4c2-485a-b3c7-094b1e42b402-kube-api-access-tt5xq\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.137664 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.137571 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-web-config\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.137985 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.137948 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e2088c6-e4c2-485a-b3c7-094b1e42b402-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.138053 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.138005 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.138102 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.138054 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.138102 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.138082 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1e2088c6-e4c2-485a-b3c7-094b1e42b402-config-out\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.138196 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.138140 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.138248 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.138192 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.138248 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.138240 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1e2088c6-e4c2-485a-b3c7-094b1e42b402-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.138370 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.138269 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e2088c6-e4c2-485a-b3c7-094b1e42b402-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.138370 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.138319 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.139273 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.138942 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e2088c6-e4c2-485a-b3c7-094b1e42b402-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.141807 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.141434 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.142572 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.142213 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1e2088c6-e4c2-485a-b3c7-094b1e42b402-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.142572 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.142243 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1e2088c6-e4c2-485a-b3c7-094b1e42b402-config-out\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.142572 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.142494 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1e2088c6-e4c2-485a-b3c7-094b1e42b402-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.143085 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.143007 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-config-volume\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.143239 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.143211 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-web-config\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.143476 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.143458 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e2088c6-e4c2-485a-b3c7-094b1e42b402-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.143535 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.143515 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.143770 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.143745 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.144222 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.144199 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.144338 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.144206 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.156278 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.153971 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt5xq\" (UniqueName: \"kubernetes.io/projected/1e2088c6-e4c2-485a-b3c7-094b1e42b402-kube-api-access-tt5xq\") pod \"alertmanager-main-0\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.175517 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.175484 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:03:06.202565 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.202530 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:03:06.202746 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.202621 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:03:06.208091 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.208065 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:03:06.240321 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.240272 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:06.406679 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.406640 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:06.443905 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.443873 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:03:06.639152 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.639119 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:06.795003 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:06.794915 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:09.286992 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:09.286961 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:09.372025 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:09.371992 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:09.590810 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:09.590691 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:10.025974 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.025941 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6664bc654c-fl9ck"] Sep 10 16:03:10.033998 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.033965 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.046292 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.046259 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 10 16:03:10.058992 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.058949 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6664bc654c-fl9ck"] Sep 10 16:03:10.078554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.078512 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1abf66ea-0b80-47d4-b51e-670b7618c703-console-serving-cert\") pod \"console-6664bc654c-fl9ck\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.078554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.078561 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-console-config\") pod \"console-6664bc654c-fl9ck\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.078796 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.078611 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-oauth-serving-cert\") pod \"console-6664bc654c-fl9ck\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.078796 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.078681 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1abf66ea-0b80-47d4-b51e-670b7618c703-console-oauth-config\") pod \"console-6664bc654c-fl9ck\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.078796 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.078747 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-service-ca\") pod \"console-6664bc654c-fl9ck\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.078796 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.078776 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-trusted-ca-bundle\") pod \"console-6664bc654c-fl9ck\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.079031 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.078863 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk9x9\" (UniqueName: \"kubernetes.io/projected/1abf66ea-0b80-47d4-b51e-670b7618c703-kube-api-access-jk9x9\") pod \"console-6664bc654c-fl9ck\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.179842 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.179802 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1abf66ea-0b80-47d4-b51e-670b7618c703-console-serving-cert\") pod \"console-6664bc654c-fl9ck\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.179842 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.179839 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-console-config\") pod \"console-6664bc654c-fl9ck\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.180086 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.179905 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-oauth-serving-cert\") pod \"console-6664bc654c-fl9ck\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.180086 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.179933 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1abf66ea-0b80-47d4-b51e-670b7618c703-console-oauth-config\") pod \"console-6664bc654c-fl9ck\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.180086 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.179978 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-service-ca\") pod \"console-6664bc654c-fl9ck\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.180086 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.180001 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-trusted-ca-bundle\") pod \"console-6664bc654c-fl9ck\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.180086 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.180038 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk9x9\" (UniqueName: \"kubernetes.io/projected/1abf66ea-0b80-47d4-b51e-670b7618c703-kube-api-access-jk9x9\") pod \"console-6664bc654c-fl9ck\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.180869 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.180773 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-oauth-serving-cert\") pod \"console-6664bc654c-fl9ck\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.180869 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.180773 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-console-config\") pod \"console-6664bc654c-fl9ck\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.180869 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.180773 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-service-ca\") pod \"console-6664bc654c-fl9ck\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.181255 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.180917 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-trusted-ca-bundle\") pod \"console-6664bc654c-fl9ck\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.182989 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.182970 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1abf66ea-0b80-47d4-b51e-670b7618c703-console-serving-cert\") pod \"console-6664bc654c-fl9ck\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.183083 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.183053 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1abf66ea-0b80-47d4-b51e-670b7618c703-console-oauth-config\") pod \"console-6664bc654c-fl9ck\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.202810 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.202783 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk9x9\" (UniqueName: \"kubernetes.io/projected/1abf66ea-0b80-47d4-b51e-670b7618c703-kube-api-access-jk9x9\") pod \"console-6664bc654c-fl9ck\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.330872 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.330800 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:03:10.349558 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.349530 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:10.640261 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.640223 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:10.714114 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:10.714079 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:11.235090 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:03:11.235053 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab59368_68c9_4b6b_a61f_6cf978e2fb30.slice/crio-f320baf3378c276868589621634b4aa97b99bd57ac7291045554adc8e2cb71c5 WatchSource:0}: Error finding container f320baf3378c276868589621634b4aa97b99bd57ac7291045554adc8e2cb71c5: Status 404 returned error can't find the container with id f320baf3378c276868589621634b4aa97b99bd57ac7291045554adc8e2cb71c5 Sep 10 16:03:11.463071 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:11.463035 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vblpz" event={"ID":"0ab59368-68c9-4b6b-a61f-6cf978e2fb30","Type":"ContainerStarted","Data":"f320baf3378c276868589621634b4aa97b99bd57ac7291045554adc8e2cb71c5"} Sep 10 16:03:11.509098 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:11.509013 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6664bc654c-fl9ck"] Sep 10 16:03:11.514712 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:11.514684 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 16:03:11.597268 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:03:11.597234 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1abf66ea_0b80_47d4_b51e_670b7618c703.slice/crio-11ea0226ed3746f84f86bbe94a957e6574caa4e1e4ca088deb395e619156ad71 WatchSource:0}: Error finding container 11ea0226ed3746f84f86bbe94a957e6574caa4e1e4ca088deb395e619156ad71: Status 404 returned error can't find the container with id 11ea0226ed3746f84f86bbe94a957e6574caa4e1e4ca088deb395e619156ad71 Sep 10 16:03:11.598055 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:03:11.598032 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e2088c6_e4c2_485a_b3c7_094b1e42b402.slice/crio-86de33cc3b8b5eb7b30e15744d0209553fff0f03fb6a1307d9934796cde9ec14 WatchSource:0}: Error finding container 86de33cc3b8b5eb7b30e15744d0209553fff0f03fb6a1307d9934796cde9ec14: Status 404 returned error can't find the container with id 86de33cc3b8b5eb7b30e15744d0209553fff0f03fb6a1307d9934796cde9ec14 Sep 10 16:03:12.473676 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:12.473613 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6664bc654c-fl9ck" event={"ID":"1abf66ea-0b80-47d4-b51e-670b7618c703","Type":"ContainerStarted","Data":"d541b1662b81affba48be26903c8ffcf73ce64babfe3468f16fa15394a141492"} Sep 10 16:03:12.473676 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:12.473671 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6664bc654c-fl9ck" event={"ID":"1abf66ea-0b80-47d4-b51e-670b7618c703","Type":"ContainerStarted","Data":"11ea0226ed3746f84f86bbe94a957e6574caa4e1e4ca088deb395e619156ad71"} Sep 10 16:03:12.477434 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:12.477363 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-vmph9" event={"ID":"03fe5d54-88f0-49bf-ba0b-96e56def25dc","Type":"ContainerStarted","Data":"cf42319feafa6d23b388260a6550e410dc3029f94a2ef64dfe61eae015910115"} Sep 10 16:03:12.480049 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:12.480019 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-vmph9" Sep 10 16:03:12.482066 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:12.482017 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e2088c6-e4c2-485a-b3c7-094b1e42b402","Type":"ContainerStarted","Data":"86de33cc3b8b5eb7b30e15744d0209553fff0f03fb6a1307d9934796cde9ec14"} Sep 10 16:03:12.488088 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:12.488037 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-vmph9" Sep 10 16:03:12.505935 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:12.505873 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6664bc654c-fl9ck" podStartSLOduration=3.505852297 podStartE2EDuration="3.505852297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:03:12.503545746 +0000 UTC m=+92.026790942" watchObservedRunningTime="2026-09-10 16:03:12.505852297 +0000 UTC m=+92.029097495" Sep 10 16:03:12.543889 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:12.541085 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-vmph9" podStartSLOduration=2.113783677 podStartE2EDuration="19.541065359s" totalImagesPullingTime="17.427281682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:03:12.539465296 +0000 UTC m=+92.062710494" watchObservedRunningTime="2026-09-10 16:03:12.541065359 +0000 UTC m=+92.064310557" Sep 10 16:03:14.296441 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:14.296352 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:14.371313 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:14.371271 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:14.492772 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:14.492735 2541 generic.go:356] "Generic (PLEG): container finished" podID="0ab59368-68c9-4b6b-a61f-6cf978e2fb30" containerID="f7c06bb93b3710877edd600cd4f6c7ec4918031ebcfdc4da4a2f2bf96e7f46e0" exitCode=0 Sep 10 16:03:14.492949 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:14.492829 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vblpz" event={"ID":"0ab59368-68c9-4b6b-a61f-6cf978e2fb30","Type":"ContainerDied","Data":"f7c06bb93b3710877edd600cd4f6c7ec4918031ebcfdc4da4a2f2bf96e7f46e0"} Sep 10 16:03:14.494820 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:14.494792 2541 generic.go:356] "Generic (PLEG): container finished" podID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerID="5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9" exitCode=0 Sep 10 16:03:14.494947 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:14.494926 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e2088c6-e4c2-485a-b3c7-094b1e42b402","Type":"ContainerDied","Data":"5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9"} Sep 10 16:03:15.434896 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:15.434857 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65b9595f8c-v7mjc"] Sep 10 16:03:15.502440 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:15.502398 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vblpz" event={"ID":"0ab59368-68c9-4b6b-a61f-6cf978e2fb30","Type":"ContainerStarted","Data":"47414505c8671cd687b6371be0834703de3ee903da082695bcee212c2c4709f9"} Sep 10 16:03:15.502628 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:15.502494 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vblpz" event={"ID":"0ab59368-68c9-4b6b-a61f-6cf978e2fb30","Type":"ContainerStarted","Data":"9c2d398d994942c1e51085c329292efe057d6b714f77f250a75d89cce414931f"} Sep 10 16:03:15.525148 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:15.525093 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vblpz" podStartSLOduration=9.036159639 podStartE2EDuration="11.525076915s" totalImagesPullingTime="2.395865535s" totalInitContainerRuntime="93.051741ms" isStatefulPod=true podCreationTimestamp="2026-09-10 16:03:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:03:15.523645943 +0000 UTC m=+95.046891141" watchObservedRunningTime="2026-09-10 16:03:15.525076915 +0000 UTC m=+95.048322111" Sep 10 16:03:15.602965 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:15.602924 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:15.698208 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:15.698010 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:15.898157 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:15.898100 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:15.978113 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:15.978009 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:16.157950 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.157915 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:16.288670 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.288575 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:16.522247 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.522219 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6664bc654c-fl9ck"] Sep 10 16:03:16.622779 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.622723 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-fbdfdfbbf-zfxjn"] Sep 10 16:03:16.638678 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.638650 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:16.657927 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.657897 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fbdfdfbbf-zfxjn"] Sep 10 16:03:16.750963 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.750920 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-trusted-ca-bundle\") pod \"console-fbdfdfbbf-zfxjn\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:16.751156 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.751007 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d9d5450-eccc-4220-9430-3dc0b43a7e65-console-oauth-config\") pod \"console-fbdfdfbbf-zfxjn\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:16.751156 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.751050 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-oauth-serving-cert\") pod \"console-fbdfdfbbf-zfxjn\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:16.751156 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.751078 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj5nn\" (UniqueName: \"kubernetes.io/projected/8d9d5450-eccc-4220-9430-3dc0b43a7e65-kube-api-access-hj5nn\") pod \"console-fbdfdfbbf-zfxjn\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:16.751156 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.751141 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-console-config\") pod \"console-fbdfdfbbf-zfxjn\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:16.751369 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.751196 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d9d5450-eccc-4220-9430-3dc0b43a7e65-console-serving-cert\") pod \"console-fbdfdfbbf-zfxjn\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:16.751369 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.751213 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-service-ca\") pod \"console-fbdfdfbbf-zfxjn\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:16.852381 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.852336 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-console-config\") pod \"console-fbdfdfbbf-zfxjn\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:16.852558 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.852449 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d9d5450-eccc-4220-9430-3dc0b43a7e65-console-serving-cert\") pod \"console-fbdfdfbbf-zfxjn\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:16.852558 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.852475 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-service-ca\") pod \"console-fbdfdfbbf-zfxjn\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:16.852558 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.852506 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-trusted-ca-bundle\") pod \"console-fbdfdfbbf-zfxjn\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:16.852712 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.852567 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d9d5450-eccc-4220-9430-3dc0b43a7e65-console-oauth-config\") pod \"console-fbdfdfbbf-zfxjn\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:16.852712 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.852610 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-oauth-serving-cert\") pod \"console-fbdfdfbbf-zfxjn\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:16.852712 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.852641 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj5nn\" (UniqueName: \"kubernetes.io/projected/8d9d5450-eccc-4220-9430-3dc0b43a7e65-kube-api-access-hj5nn\") pod \"console-fbdfdfbbf-zfxjn\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:16.853209 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.853181 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-console-config\") pod \"console-fbdfdfbbf-zfxjn\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:16.854006 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.853983 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-service-ca\") pod \"console-fbdfdfbbf-zfxjn\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:16.854650 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.854619 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-oauth-serving-cert\") pod \"console-fbdfdfbbf-zfxjn\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:16.854773 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.854683 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-trusted-ca-bundle\") pod \"console-fbdfdfbbf-zfxjn\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:16.855660 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.855637 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d9d5450-eccc-4220-9430-3dc0b43a7e65-console-serving-cert\") pod \"console-fbdfdfbbf-zfxjn\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:16.855757 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.855679 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d9d5450-eccc-4220-9430-3dc0b43a7e65-console-oauth-config\") pod \"console-fbdfdfbbf-zfxjn\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:16.862747 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.862708 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj5nn\" (UniqueName: \"kubernetes.io/projected/8d9d5450-eccc-4220-9430-3dc0b43a7e65-kube-api-access-hj5nn\") pod \"console-fbdfdfbbf-zfxjn\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:16.950415 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:16.950336 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:17.356817 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:17.356779 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fbdfdfbbf-zfxjn"] Sep 10 16:03:17.360509 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:03:17.360480 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9d5450_eccc_4220_9430_3dc0b43a7e65.slice/crio-cb523e353f52d8d518e418ff425c7d0540902df7d21fea924c88754cfec36f16 WatchSource:0}: Error finding container cb523e353f52d8d518e418ff425c7d0540902df7d21fea924c88754cfec36f16: Status 404 returned error can't find the container with id cb523e353f52d8d518e418ff425c7d0540902df7d21fea924c88754cfec36f16 Sep 10 16:03:17.515542 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:17.515505 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e2088c6-e4c2-485a-b3c7-094b1e42b402","Type":"ContainerStarted","Data":"195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929"} Sep 10 16:03:17.515720 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:17.515592 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e2088c6-e4c2-485a-b3c7-094b1e42b402","Type":"ContainerStarted","Data":"5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58"} Sep 10 16:03:17.515720 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:17.515631 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e2088c6-e4c2-485a-b3c7-094b1e42b402","Type":"ContainerStarted","Data":"71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3"} Sep 10 16:03:17.517555 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:17.517529 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbdfdfbbf-zfxjn" event={"ID":"8d9d5450-eccc-4220-9430-3dc0b43a7e65","Type":"ContainerStarted","Data":"8c20761464a1e55673ba4be3cfcbc4867cc0f61a414056bf5b78221a6d9e28fb"} Sep 10 16:03:17.517674 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:17.517570 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbdfdfbbf-zfxjn" event={"ID":"8d9d5450-eccc-4220-9430-3dc0b43a7e65","Type":"ContainerStarted","Data":"cb523e353f52d8d518e418ff425c7d0540902df7d21fea924c88754cfec36f16"} Sep 10 16:03:17.540643 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:17.540596 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-fbdfdfbbf-zfxjn" podStartSLOduration=1.54058169 podStartE2EDuration="1.54058169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:03:17.539092732 +0000 UTC m=+97.062337951" watchObservedRunningTime="2026-09-10 16:03:17.54058169 +0000 UTC m=+97.063826888" Sep 10 16:03:18.526888 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:18.526855 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e2088c6-e4c2-485a-b3c7-094b1e42b402","Type":"ContainerStarted","Data":"27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c"} Sep 10 16:03:18.527133 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:18.526964 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e2088c6-e4c2-485a-b3c7-094b1e42b402","Type":"ContainerStarted","Data":"84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548"} Sep 10 16:03:19.535836 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:19.535746 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e2088c6-e4c2-485a-b3c7-094b1e42b402","Type":"ContainerStarted","Data":"d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04"} Sep 10 16:03:19.581127 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:19.581065 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=8.412354079 podStartE2EDuration="14.581046953s" totalImagesPullingTime="6.149901399s" totalInitContainerRuntime="18.791475ms" isStatefulPod=false podCreationTimestamp="2026-09-10 16:03:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:03:19.57868325 +0000 UTC m=+99.101928472" watchObservedRunningTime="2026-09-10 16:03:19.581046953 +0000 UTC m=+99.104292150" Sep 10 16:03:20.308515 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:20.308474 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:20.350728 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:20.350688 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:20.432499 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:20.432323 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:23.053855 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:23.053817 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:23.182462 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:23.182426 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:24.358500 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:24.358467 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j2m6r" Sep 10 16:03:25.371551 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:25.371516 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:25.520393 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:25.520367 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:26.950756 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:26.950723 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:26.951222 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:26.950796 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:26.955396 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:26.955376 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:27.572719 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:27.572690 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:03:27.649606 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:27.649577 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6459747688-mgvzg"] Sep 10 16:03:27.840916 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:27.840838 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:27.940326 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:27.940264 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:33.594255 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:33.594186 2541 generic.go:356] "Generic (PLEG): container finished" podID="3a5e2ec6-0e23-4d13-a153-25d193930e01" containerID="3fa58b942592bacbb8fa880bddafec80507e18d402e9038be3938744f9d0c72c" exitCode=0 Sep 10 16:03:33.594747 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:33.594258 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" event={"ID":"3a5e2ec6-0e23-4d13-a153-25d193930e01","Type":"ContainerDied","Data":"3fa58b942592bacbb8fa880bddafec80507e18d402e9038be3938744f9d0c72c"} Sep 10 16:03:33.594747 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:33.594639 2541 scope.go:117] "RemoveContainer" containerID="3fa58b942592bacbb8fa880bddafec80507e18d402e9038be3938744f9d0c72c" Sep 10 16:03:34.599752 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:34.599716 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-89xqs" event={"ID":"3a5e2ec6-0e23-4d13-a153-25d193930e01","Type":"ContainerStarted","Data":"323a3ba2527ef0573b7370ae66f5888e5fff93e8acb343e9931baff762c8bcfc"} Sep 10 16:03:38.028995 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:38.028959 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:38.099884 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:38.099851 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:40.458631 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.458591 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" podUID="13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad" containerName="registry" containerID="cri-o://7d409c6a977115acc3c1f72eea4420b65e2b5733b5f096882a076d79614da17d" gracePeriod=30 Sep 10 16:03:40.625973 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.625941 2541 generic.go:356] "Generic (PLEG): container finished" podID="13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad" containerID="7d409c6a977115acc3c1f72eea4420b65e2b5733b5f096882a076d79614da17d" exitCode=0 Sep 10 16:03:40.626169 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.626017 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" event={"ID":"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad","Type":"ContainerDied","Data":"7d409c6a977115acc3c1f72eea4420b65e2b5733b5f096882a076d79614da17d"} Sep 10 16:03:40.736426 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.736372 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:03:40.877609 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.877578 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-ca-trust-extracted\") pod \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " Sep 10 16:03:40.877758 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.877625 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-bound-sa-token\") pod \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " Sep 10 16:03:40.877758 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.877668 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-certificates\") pod \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " Sep 10 16:03:40.877758 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.877726 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-image-registry-private-configuration\") pod \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " Sep 10 16:03:40.877758 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.877755 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-installation-pull-secrets\") pod \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " Sep 10 16:03:40.877959 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.877784 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls\") pod \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " Sep 10 16:03:40.877959 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.877947 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-trusted-ca\") pod \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " Sep 10 16:03:40.878054 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.877992 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbp48\" (UniqueName: \"kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-kube-api-access-dbp48\") pod \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\" (UID: \"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad\") " Sep 10 16:03:40.878370 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.878336 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad" (UID: "13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:03:40.878646 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.878427 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad" (UID: "13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:03:40.880389 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.880365 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad" (UID: "13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:03:40.880497 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.880470 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad" (UID: "13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:03:40.880648 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.880624 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad" (UID: "13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:03:40.880889 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.880870 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad" (UID: "13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:03:40.880955 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.880938 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-kube-api-access-dbp48" (OuterVolumeSpecName: "kube-api-access-dbp48") pod "13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad" (UID: "13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad"). InnerVolumeSpecName "kube-api-access-dbp48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:03:40.886350 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.886322 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad" (UID: "13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:03:40.979621 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.979587 2541 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-ca-trust-extracted\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:03:40.979621 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.979620 2541 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-bound-sa-token\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:03:40.979866 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.979641 2541 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-certificates\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:03:40.979866 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.979657 2541 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-image-registry-private-configuration\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:03:40.979866 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.979787 2541 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-installation-pull-secrets\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:03:40.979866 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.979801 2541 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-registry-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:03:40.979866 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.979819 2541 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-trusted-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:03:40.980185 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:40.979833 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbp48\" (UniqueName: \"kubernetes.io/projected/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad-kube-api-access-dbp48\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:03:41.549580 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:41.549536 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6664bc654c-fl9ck" podUID="1abf66ea-0b80-47d4-b51e-670b7618c703" containerName="console" containerID="cri-o://d541b1662b81affba48be26903c8ffcf73ce64babfe3468f16fa15394a141492" gracePeriod=15 Sep 10 16:03:41.631739 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:41.631699 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" event={"ID":"13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad","Type":"ContainerDied","Data":"51f5b5bd835ccec1f7b7fe0b6a23c4dc2595bd57e4a818a3faa0c60cf0a02b96"} Sep 10 16:03:41.631908 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:41.631753 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65b9595f8c-v7mjc" Sep 10 16:03:41.631908 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:41.631807 2541 scope.go:117] "RemoveContainer" containerID="7d409c6a977115acc3c1f72eea4420b65e2b5733b5f096882a076d79614da17d" Sep 10 16:03:41.664880 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:41.664854 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65b9595f8c-v7mjc"] Sep 10 16:03:41.673566 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:41.673540 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-65b9595f8c-v7mjc"] Sep 10 16:03:41.858262 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:41.858229 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:41.987219 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:41.987184 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jk9x9\" (UniqueName: \"kubernetes.io/projected/1abf66ea-0b80-47d4-b51e-670b7618c703-kube-api-access-jk9x9\") pod \"1abf66ea-0b80-47d4-b51e-670b7618c703\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " Sep 10 16:03:41.987219 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:41.987220 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-service-ca\") pod \"1abf66ea-0b80-47d4-b51e-670b7618c703\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " Sep 10 16:03:41.987503 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:41.987254 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1abf66ea-0b80-47d4-b51e-670b7618c703-console-oauth-config\") pod \"1abf66ea-0b80-47d4-b51e-670b7618c703\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " Sep 10 16:03:41.987503 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:41.987293 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1abf66ea-0b80-47d4-b51e-670b7618c703-console-serving-cert\") pod \"1abf66ea-0b80-47d4-b51e-670b7618c703\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " Sep 10 16:03:41.987503 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:41.987397 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-trusted-ca-bundle\") pod \"1abf66ea-0b80-47d4-b51e-670b7618c703\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " Sep 10 16:03:41.987503 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:41.987461 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-oauth-serving-cert\") pod \"1abf66ea-0b80-47d4-b51e-670b7618c703\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " Sep 10 16:03:41.987699 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:41.987513 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-console-config\") pod \"1abf66ea-0b80-47d4-b51e-670b7618c703\" (UID: \"1abf66ea-0b80-47d4-b51e-670b7618c703\") " Sep 10 16:03:41.987795 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:41.987768 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-service-ca" (OuterVolumeSpecName: "service-ca") pod "1abf66ea-0b80-47d4-b51e-670b7618c703" (UID: "1abf66ea-0b80-47d4-b51e-670b7618c703"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:03:41.987856 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:41.987794 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1abf66ea-0b80-47d4-b51e-670b7618c703" (UID: "1abf66ea-0b80-47d4-b51e-670b7618c703"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:03:41.987856 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:41.987839 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1abf66ea-0b80-47d4-b51e-670b7618c703" (UID: "1abf66ea-0b80-47d4-b51e-670b7618c703"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:03:41.988109 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:41.988055 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-console-config" (OuterVolumeSpecName: "console-config") pod "1abf66ea-0b80-47d4-b51e-670b7618c703" (UID: "1abf66ea-0b80-47d4-b51e-670b7618c703"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:03:41.989699 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:41.989678 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1abf66ea-0b80-47d4-b51e-670b7618c703-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1abf66ea-0b80-47d4-b51e-670b7618c703" (UID: "1abf66ea-0b80-47d4-b51e-670b7618c703"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:03:41.989796 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:41.989732 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1abf66ea-0b80-47d4-b51e-670b7618c703-kube-api-access-jk9x9" (OuterVolumeSpecName: "kube-api-access-jk9x9") pod "1abf66ea-0b80-47d4-b51e-670b7618c703" (UID: "1abf66ea-0b80-47d4-b51e-670b7618c703"). InnerVolumeSpecName "kube-api-access-jk9x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:03:41.989796 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:41.989743 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1abf66ea-0b80-47d4-b51e-670b7618c703-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1abf66ea-0b80-47d4-b51e-670b7618c703" (UID: "1abf66ea-0b80-47d4-b51e-670b7618c703"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:03:42.088791 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:42.088699 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk9x9\" (UniqueName: \"kubernetes.io/projected/1abf66ea-0b80-47d4-b51e-670b7618c703-kube-api-access-jk9x9\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:03:42.088791 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:42.088732 2541 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:03:42.088791 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:42.088741 2541 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1abf66ea-0b80-47d4-b51e-670b7618c703-console-oauth-config\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:03:42.088791 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:42.088753 2541 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1abf66ea-0b80-47d4-b51e-670b7618c703-console-serving-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:03:42.088791 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:42.088760 2541 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-trusted-ca-bundle\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:03:42.088791 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:42.088768 2541 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-oauth-serving-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:03:42.088791 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:42.088777 2541 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1abf66ea-0b80-47d4-b51e-670b7618c703-console-config\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:03:42.637226 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:42.637186 2541 generic.go:356] "Generic (PLEG): container finished" podID="1abf66ea-0b80-47d4-b51e-670b7618c703" containerID="d541b1662b81affba48be26903c8ffcf73ce64babfe3468f16fa15394a141492" exitCode=0 Sep 10 16:03:42.637680 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:42.637237 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6664bc654c-fl9ck" Sep 10 16:03:42.637680 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:42.637275 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6664bc654c-fl9ck" event={"ID":"1abf66ea-0b80-47d4-b51e-670b7618c703","Type":"ContainerDied","Data":"d541b1662b81affba48be26903c8ffcf73ce64babfe3468f16fa15394a141492"} Sep 10 16:03:42.637680 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:42.637356 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6664bc654c-fl9ck" event={"ID":"1abf66ea-0b80-47d4-b51e-670b7618c703","Type":"ContainerDied","Data":"11ea0226ed3746f84f86bbe94a957e6574caa4e1e4ca088deb395e619156ad71"} Sep 10 16:03:42.637680 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:42.637387 2541 scope.go:117] "RemoveContainer" containerID="d541b1662b81affba48be26903c8ffcf73ce64babfe3468f16fa15394a141492" Sep 10 16:03:42.646690 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:42.646673 2541 scope.go:117] "RemoveContainer" containerID="d541b1662b81affba48be26903c8ffcf73ce64babfe3468f16fa15394a141492" Sep 10 16:03:42.647015 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:03:42.646993 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d541b1662b81affba48be26903c8ffcf73ce64babfe3468f16fa15394a141492\": container with ID starting with d541b1662b81affba48be26903c8ffcf73ce64babfe3468f16fa15394a141492 not found: ID does not exist" containerID="d541b1662b81affba48be26903c8ffcf73ce64babfe3468f16fa15394a141492" Sep 10 16:03:42.647071 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:42.647024 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d541b1662b81affba48be26903c8ffcf73ce64babfe3468f16fa15394a141492"} err="failed to get container status \"d541b1662b81affba48be26903c8ffcf73ce64babfe3468f16fa15394a141492\": rpc error: code = NotFound desc = could not find container \"d541b1662b81affba48be26903c8ffcf73ce64babfe3468f16fa15394a141492\": container with ID starting with d541b1662b81affba48be26903c8ffcf73ce64babfe3468f16fa15394a141492 not found: ID does not exist" Sep 10 16:03:42.660582 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:42.660542 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6664bc654c-fl9ck"] Sep 10 16:03:42.663794 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:42.663766 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6664bc654c-fl9ck"] Sep 10 16:03:42.977804 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:42.977730 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad" path="/var/lib/kubelet/pods/13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad/volumes" Sep 10 16:03:42.978120 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:42.978107 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1abf66ea-0b80-47d4-b51e-670b7618c703" path="/var/lib/kubelet/pods/1abf66ea-0b80-47d4-b51e-670b7618c703/volumes" Sep 10 16:03:50.960853 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:50.960813 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs\") pod \"network-metrics-daemon-h5frk\" (UID: \"086a5470-5fac-4be6-963c-7b219b25b740\") " pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:03:50.965093 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:50.964686 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086a5470-5fac-4be6-963c-7b219b25b740-metrics-certs\") pod \"network-metrics-daemon-h5frk\" (UID: \"086a5470-5fac-4be6-963c-7b219b25b740\") " pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:03:50.990871 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:50.990842 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-swxfx\"" Sep 10 16:03:50.998117 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:50.998094 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5frk" Sep 10 16:03:51.094006 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:51.093976 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:51.187528 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:51.187487 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:51.308680 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:51.308480 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h5frk"] Sep 10 16:03:51.311371 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:03:51.311342 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086a5470_5fac_4be6_963c_7b219b25b740.slice/crio-50445cc065dfadbe99ee46cdd497160eafb95bd6fea9c2b40472188fcca03ee3 WatchSource:0}: Error finding container 50445cc065dfadbe99ee46cdd497160eafb95bd6fea9c2b40472188fcca03ee3: Status 404 returned error can't find the container with id 50445cc065dfadbe99ee46cdd497160eafb95bd6fea9c2b40472188fcca03ee3 Sep 10 16:03:51.682629 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:51.682597 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h5frk" event={"ID":"086a5470-5fac-4be6-963c-7b219b25b740","Type":"ContainerStarted","Data":"50445cc065dfadbe99ee46cdd497160eafb95bd6fea9c2b40472188fcca03ee3"} Sep 10 16:03:52.670899 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:52.670859 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6459747688-mgvzg" podUID="59656fa7-1ec7-40d1-bbbc-73927c3f4ff6" containerName="console" containerID="cri-o://bee3170929383d671753ca8ec599354a4258eda3b810b4f5cbf513afef60b6d4" gracePeriod=15 Sep 10 16:03:52.994588 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:52.994556 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:03:53.077468 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.077427 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-console-oauth-config\") pod \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " Sep 10 16:03:53.077668 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.077481 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-service-ca\") pod \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " Sep 10 16:03:53.077668 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.077535 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-console-config\") pod \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " Sep 10 16:03:53.077668 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.077586 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-console-serving-cert\") pod \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " Sep 10 16:03:53.077668 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.077619 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-oauth-serving-cert\") pod \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " Sep 10 16:03:53.077668 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.077642 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb2vx\" (UniqueName: \"kubernetes.io/projected/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-kube-api-access-rb2vx\") pod \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\" (UID: \"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6\") " Sep 10 16:03:53.078078 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.078043 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-console-config" (OuterVolumeSpecName: "console-config") pod "59656fa7-1ec7-40d1-bbbc-73927c3f4ff6" (UID: "59656fa7-1ec7-40d1-bbbc-73927c3f4ff6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:03:53.078240 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.078049 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-service-ca" (OuterVolumeSpecName: "service-ca") pod "59656fa7-1ec7-40d1-bbbc-73927c3f4ff6" (UID: "59656fa7-1ec7-40d1-bbbc-73927c3f4ff6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:03:53.078240 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.078080 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "59656fa7-1ec7-40d1-bbbc-73927c3f4ff6" (UID: "59656fa7-1ec7-40d1-bbbc-73927c3f4ff6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:03:53.079907 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.079884 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "59656fa7-1ec7-40d1-bbbc-73927c3f4ff6" (UID: "59656fa7-1ec7-40d1-bbbc-73927c3f4ff6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:03:53.079997 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.079903 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "59656fa7-1ec7-40d1-bbbc-73927c3f4ff6" (UID: "59656fa7-1ec7-40d1-bbbc-73927c3f4ff6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:03:53.079997 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.079954 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-kube-api-access-rb2vx" (OuterVolumeSpecName: "kube-api-access-rb2vx") pod "59656fa7-1ec7-40d1-bbbc-73927c3f4ff6" (UID: "59656fa7-1ec7-40d1-bbbc-73927c3f4ff6"). InnerVolumeSpecName "kube-api-access-rb2vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:03:53.178960 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.178927 2541 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-console-config\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:03:53.178960 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.178954 2541 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-console-serving-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:03:53.178960 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.178964 2541 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-oauth-serving-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:03:53.179191 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.178972 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb2vx\" (UniqueName: \"kubernetes.io/projected/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-kube-api-access-rb2vx\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:03:53.179191 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.178981 2541 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-console-oauth-config\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:03:53.179191 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.178990 2541 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:03:53.693172 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.693128 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h5frk" event={"ID":"086a5470-5fac-4be6-963c-7b219b25b740","Type":"ContainerStarted","Data":"0c4b9439259ee6bf08bf13b316c7b3529a418fb92a7d8892df3911b8914be984"} Sep 10 16:03:53.693603 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.693204 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h5frk" event={"ID":"086a5470-5fac-4be6-963c-7b219b25b740","Type":"ContainerStarted","Data":"d15b7613674c5179172c513c42a2c5ce4bd73a59dc4e6b0ed7fd0aa0033e9430"} Sep 10 16:03:53.694458 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.694431 2541 generic.go:356] "Generic (PLEG): container finished" podID="59656fa7-1ec7-40d1-bbbc-73927c3f4ff6" containerID="bee3170929383d671753ca8ec599354a4258eda3b810b4f5cbf513afef60b6d4" exitCode=0 Sep 10 16:03:53.694573 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.694467 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6459747688-mgvzg" Sep 10 16:03:53.694573 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.694512 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6459747688-mgvzg" event={"ID":"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6","Type":"ContainerDied","Data":"bee3170929383d671753ca8ec599354a4258eda3b810b4f5cbf513afef60b6d4"} Sep 10 16:03:53.694573 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.694550 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6459747688-mgvzg" event={"ID":"59656fa7-1ec7-40d1-bbbc-73927c3f4ff6","Type":"ContainerDied","Data":"3a330319508d5db0337f96c623f37f27d743118c70d8f03a7172bc1e17158d57"} Sep 10 16:03:53.694573 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.694571 2541 scope.go:117] "RemoveContainer" containerID="bee3170929383d671753ca8ec599354a4258eda3b810b4f5cbf513afef60b6d4" Sep 10 16:03:53.704681 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.704183 2541 scope.go:117] "RemoveContainer" containerID="bee3170929383d671753ca8ec599354a4258eda3b810b4f5cbf513afef60b6d4" Sep 10 16:03:53.704681 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:03:53.704560 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bee3170929383d671753ca8ec599354a4258eda3b810b4f5cbf513afef60b6d4\": container with ID starting with bee3170929383d671753ca8ec599354a4258eda3b810b4f5cbf513afef60b6d4 not found: ID does not exist" containerID="bee3170929383d671753ca8ec599354a4258eda3b810b4f5cbf513afef60b6d4" Sep 10 16:03:53.704681 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.704599 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bee3170929383d671753ca8ec599354a4258eda3b810b4f5cbf513afef60b6d4"} err="failed to get container status \"bee3170929383d671753ca8ec599354a4258eda3b810b4f5cbf513afef60b6d4\": rpc error: code = NotFound desc = could not find container \"bee3170929383d671753ca8ec599354a4258eda3b810b4f5cbf513afef60b6d4\": container with ID starting with bee3170929383d671753ca8ec599354a4258eda3b810b4f5cbf513afef60b6d4 not found: ID does not exist" Sep 10 16:03:53.711847 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.711790 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-h5frk" podStartSLOduration=130.327720442 podStartE2EDuration="2m11.711776585s" totalImagesPullingTime="1.384056143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:03:53.710036088 +0000 UTC m=+133.233281284" watchObservedRunningTime="2026-09-10 16:03:53.711776585 +0000 UTC m=+133.235021782" Sep 10 16:03:53.727167 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.727134 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6459747688-mgvzg"] Sep 10 16:03:53.731437 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.731409 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6459747688-mgvzg"] Sep 10 16:03:53.834237 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.834201 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:53.920632 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:53.920591 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:54.978345 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:03:54.978291 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59656fa7-1ec7-40d1-bbbc-73927c3f4ff6" path="/var/lib/kubelet/pods/59656fa7-1ec7-40d1-bbbc-73927c3f4ff6/volumes" Sep 10 16:05:04.704247 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:05:04.704210 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:06.661731 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:05:06.661698 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:12.488347 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:05:12.488294 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:30.922986 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:06:30.922948 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:40.911774 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:06:40.911747 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/1.log" Sep 10 16:06:40.916949 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:06:40.916926 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/1.log" Sep 10 16:06:40.918006 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:06:40.917988 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4c6j_2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd/ovn-acl-logging/0.log" Sep 10 16:06:40.921640 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:06:40.921623 2541 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 16:06:40.923603 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:06:40.923587 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4c6j_2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd/ovn-acl-logging/0.log" Sep 10 16:07:07.971400 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:07:07.971367 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:10.721081 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:10.721046 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 16:08:10.721669 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:10.721432 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="kube-rbac-proxy-metric" containerID="cri-o://27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c" gracePeriod=120 Sep 10 16:08:10.721669 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:10.721432 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="alertmanager" containerID="cri-o://71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3" gracePeriod=120 Sep 10 16:08:10.721669 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:10.721602 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="prom-label-proxy" containerID="cri-o://d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04" gracePeriod=120 Sep 10 16:08:10.721669 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:10.721437 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="kube-rbac-proxy-web" containerID="cri-o://195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929" gracePeriod=120 Sep 10 16:08:10.721881 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:10.721691 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="kube-rbac-proxy" containerID="cri-o://84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548" gracePeriod=120 Sep 10 16:08:10.721881 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:10.721444 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="config-reloader" containerID="cri-o://5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58" gracePeriod=120 Sep 10 16:08:10.978418 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:10.978355 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:11.121821 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.121786 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1e2088c6-e4c2-485a-b3c7-094b1e42b402-alertmanager-main-db\") pod \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " Sep 10 16:08:11.121821 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.121826 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tt5xq\" (UniqueName: \"kubernetes.io/projected/1e2088c6-e4c2-485a-b3c7-094b1e42b402-kube-api-access-tt5xq\") pod \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " Sep 10 16:08:11.122075 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.121850 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-web-config\") pod \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " Sep 10 16:08:11.122075 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.121877 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e2088c6-e4c2-485a-b3c7-094b1e42b402-alertmanager-trusted-ca-bundle\") pod \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " Sep 10 16:08:11.122075 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.121916 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-cluster-tls-config\") pod \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " Sep 10 16:08:11.122075 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.121943 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-main-tls\") pod \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " Sep 10 16:08:11.122075 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.121998 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-kube-rbac-proxy\") pod \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " Sep 10 16:08:11.122075 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.122023 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-config-volume\") pod \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " Sep 10 16:08:11.122075 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.122066 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-kube-rbac-proxy-web\") pod \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " Sep 10 16:08:11.122525 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.122112 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " Sep 10 16:08:11.122525 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.122135 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1e2088c6-e4c2-485a-b3c7-094b1e42b402-tls-assets\") pod \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " Sep 10 16:08:11.122525 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.122161 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1e2088c6-e4c2-485a-b3c7-094b1e42b402-config-out\") pod \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " Sep 10 16:08:11.122525 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.122137 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2088c6-e4c2-485a-b3c7-094b1e42b402-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "1e2088c6-e4c2-485a-b3c7-094b1e42b402" (UID: "1e2088c6-e4c2-485a-b3c7-094b1e42b402"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:08:11.122525 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.122191 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e2088c6-e4c2-485a-b3c7-094b1e42b402-metrics-client-ca\") pod \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\" (UID: \"1e2088c6-e4c2-485a-b3c7-094b1e42b402\") " Sep 10 16:08:11.122525 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.122361 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2088c6-e4c2-485a-b3c7-094b1e42b402-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "1e2088c6-e4c2-485a-b3c7-094b1e42b402" (UID: "1e2088c6-e4c2-485a-b3c7-094b1e42b402"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:08:11.122525 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.122597 2541 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e2088c6-e4c2-485a-b3c7-094b1e42b402-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:08:11.123041 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.122618 2541 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1e2088c6-e4c2-485a-b3c7-094b1e42b402-alertmanager-main-db\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:08:11.123368 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.123336 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2088c6-e4c2-485a-b3c7-094b1e42b402-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "1e2088c6-e4c2-485a-b3c7-094b1e42b402" (UID: "1e2088c6-e4c2-485a-b3c7-094b1e42b402"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:08:11.125262 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.125225 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2088c6-e4c2-485a-b3c7-094b1e42b402-kube-api-access-tt5xq" (OuterVolumeSpecName: "kube-api-access-tt5xq") pod "1e2088c6-e4c2-485a-b3c7-094b1e42b402" (UID: "1e2088c6-e4c2-485a-b3c7-094b1e42b402"). InnerVolumeSpecName "kube-api-access-tt5xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:08:11.125771 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.125740 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-config-volume" (OuterVolumeSpecName: "config-volume") pod "1e2088c6-e4c2-485a-b3c7-094b1e42b402" (UID: "1e2088c6-e4c2-485a-b3c7-094b1e42b402"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:08:11.126027 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.126001 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2088c6-e4c2-485a-b3c7-094b1e42b402-config-out" (OuterVolumeSpecName: "config-out") pod "1e2088c6-e4c2-485a-b3c7-094b1e42b402" (UID: "1e2088c6-e4c2-485a-b3c7-094b1e42b402"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:08:11.126107 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.126056 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "1e2088c6-e4c2-485a-b3c7-094b1e42b402" (UID: "1e2088c6-e4c2-485a-b3c7-094b1e42b402"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:08:11.126174 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.126111 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "1e2088c6-e4c2-485a-b3c7-094b1e42b402" (UID: "1e2088c6-e4c2-485a-b3c7-094b1e42b402"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:08:11.126688 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.126667 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2088c6-e4c2-485a-b3c7-094b1e42b402-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1e2088c6-e4c2-485a-b3c7-094b1e42b402" (UID: "1e2088c6-e4c2-485a-b3c7-094b1e42b402"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:08:11.127017 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.127002 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "1e2088c6-e4c2-485a-b3c7-094b1e42b402" (UID: "1e2088c6-e4c2-485a-b3c7-094b1e42b402"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:08:11.127281 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.127266 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "1e2088c6-e4c2-485a-b3c7-094b1e42b402" (UID: "1e2088c6-e4c2-485a-b3c7-094b1e42b402"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:08:11.129848 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.129824 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "1e2088c6-e4c2-485a-b3c7-094b1e42b402" (UID: "1e2088c6-e4c2-485a-b3c7-094b1e42b402"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:08:11.136323 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.136275 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-web-config" (OuterVolumeSpecName: "web-config") pod "1e2088c6-e4c2-485a-b3c7-094b1e42b402" (UID: "1e2088c6-e4c2-485a-b3c7-094b1e42b402"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:08:11.222970 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.222943 2541 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:08:11.222970 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.222969 2541 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1e2088c6-e4c2-485a-b3c7-094b1e42b402-tls-assets\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:08:11.223130 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.222979 2541 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1e2088c6-e4c2-485a-b3c7-094b1e42b402-config-out\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:08:11.223130 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.222987 2541 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e2088c6-e4c2-485a-b3c7-094b1e42b402-metrics-client-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:08:11.223130 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.222996 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt5xq\" (UniqueName: \"kubernetes.io/projected/1e2088c6-e4c2-485a-b3c7-094b1e42b402-kube-api-access-tt5xq\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:08:11.223130 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.223003 2541 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-web-config\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:08:11.223130 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.223011 2541 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-cluster-tls-config\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:08:11.223130 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.223019 2541 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-main-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:08:11.223130 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.223027 2541 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:08:11.223130 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.223036 2541 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-config-volume\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:08:11.223130 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.223045 2541 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e2088c6-e4c2-485a-b3c7-094b1e42b402-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:08:11.667245 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.667212 2541 generic.go:356] "Generic (PLEG): container finished" podID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerID="d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04" exitCode=0 Sep 10 16:08:11.667245 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.667238 2541 generic.go:356] "Generic (PLEG): container finished" podID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerID="27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c" exitCode=0 Sep 10 16:08:11.667245 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.667246 2541 generic.go:356] "Generic (PLEG): container finished" podID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerID="84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548" exitCode=0 Sep 10 16:08:11.667245 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.667254 2541 generic.go:356] "Generic (PLEG): container finished" podID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerID="195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929" exitCode=0 Sep 10 16:08:11.667548 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.667261 2541 generic.go:356] "Generic (PLEG): container finished" podID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerID="5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58" exitCode=0 Sep 10 16:08:11.667548 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.667268 2541 generic.go:356] "Generic (PLEG): container finished" podID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerID="71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3" exitCode=0 Sep 10 16:08:11.667548 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.667288 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:11.667548 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.667312 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e2088c6-e4c2-485a-b3c7-094b1e42b402","Type":"ContainerDied","Data":"d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04"} Sep 10 16:08:11.667548 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.667379 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e2088c6-e4c2-485a-b3c7-094b1e42b402","Type":"ContainerDied","Data":"27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c"} Sep 10 16:08:11.667548 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.667407 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e2088c6-e4c2-485a-b3c7-094b1e42b402","Type":"ContainerDied","Data":"84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548"} Sep 10 16:08:11.667548 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.667431 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e2088c6-e4c2-485a-b3c7-094b1e42b402","Type":"ContainerDied","Data":"195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929"} Sep 10 16:08:11.667548 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.667455 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e2088c6-e4c2-485a-b3c7-094b1e42b402","Type":"ContainerDied","Data":"5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58"} Sep 10 16:08:11.667548 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.667448 2541 scope.go:117] "RemoveContainer" containerID="d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04" Sep 10 16:08:11.667548 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.667478 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e2088c6-e4c2-485a-b3c7-094b1e42b402","Type":"ContainerDied","Data":"71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3"} Sep 10 16:08:11.667980 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.667579 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e2088c6-e4c2-485a-b3c7-094b1e42b402","Type":"ContainerDied","Data":"86de33cc3b8b5eb7b30e15744d0209553fff0f03fb6a1307d9934796cde9ec14"} Sep 10 16:08:11.675758 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.675738 2541 scope.go:117] "RemoveContainer" containerID="27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c" Sep 10 16:08:11.682764 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.682750 2541 scope.go:117] "RemoveContainer" containerID="84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548" Sep 10 16:08:11.689106 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.689090 2541 scope.go:117] "RemoveContainer" containerID="195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929" Sep 10 16:08:11.692185 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.692165 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 16:08:11.695677 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.695659 2541 scope.go:117] "RemoveContainer" containerID="5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58" Sep 10 16:08:11.702065 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.702043 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 16:08:11.704419 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.704400 2541 scope.go:117] "RemoveContainer" containerID="71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3" Sep 10 16:08:11.711226 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.711211 2541 scope.go:117] "RemoveContainer" containerID="5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9" Sep 10 16:08:11.717536 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.717521 2541 scope.go:117] "RemoveContainer" containerID="d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04" Sep 10 16:08:11.717768 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:08:11.717753 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04\": container with ID starting with d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04 not found: ID does not exist" containerID="d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04" Sep 10 16:08:11.717823 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.717776 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04"} err="failed to get container status \"d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04\": rpc error: code = NotFound desc = could not find container \"d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04\": container with ID starting with d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04 not found: ID does not exist" Sep 10 16:08:11.717823 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.717791 2541 scope.go:117] "RemoveContainer" containerID="27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c" Sep 10 16:08:11.718036 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:08:11.718022 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c\": container with ID starting with 27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c not found: ID does not exist" containerID="27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c" Sep 10 16:08:11.718082 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.718039 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c"} err="failed to get container status \"27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c\": rpc error: code = NotFound desc = could not find container \"27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c\": container with ID starting with 27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c not found: ID does not exist" Sep 10 16:08:11.718082 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.718054 2541 scope.go:117] "RemoveContainer" containerID="84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548" Sep 10 16:08:11.718258 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:08:11.718240 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548\": container with ID starting with 84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548 not found: ID does not exist" containerID="84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548" Sep 10 16:08:11.718290 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.718263 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548"} err="failed to get container status \"84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548\": rpc error: code = NotFound desc = could not find container \"84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548\": container with ID starting with 84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548 not found: ID does not exist" Sep 10 16:08:11.718290 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.718278 2541 scope.go:117] "RemoveContainer" containerID="195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929" Sep 10 16:08:11.718549 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:08:11.718531 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929\": container with ID starting with 195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929 not found: ID does not exist" containerID="195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929" Sep 10 16:08:11.718601 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.718553 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929"} err="failed to get container status \"195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929\": rpc error: code = NotFound desc = could not find container \"195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929\": container with ID starting with 195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929 not found: ID does not exist" Sep 10 16:08:11.718601 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.718569 2541 scope.go:117] "RemoveContainer" containerID="5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58" Sep 10 16:08:11.718815 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:08:11.718798 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58\": container with ID starting with 5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58 not found: ID does not exist" containerID="5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58" Sep 10 16:08:11.718877 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.718823 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58"} err="failed to get container status \"5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58\": rpc error: code = NotFound desc = could not find container \"5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58\": container with ID starting with 5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58 not found: ID does not exist" Sep 10 16:08:11.718877 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.718845 2541 scope.go:117] "RemoveContainer" containerID="71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3" Sep 10 16:08:11.719081 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:08:11.719066 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3\": container with ID starting with 71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3 not found: ID does not exist" containerID="71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3" Sep 10 16:08:11.719121 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.719086 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3"} err="failed to get container status \"71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3\": rpc error: code = NotFound desc = could not find container \"71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3\": container with ID starting with 71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3 not found: ID does not exist" Sep 10 16:08:11.719121 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.719099 2541 scope.go:117] "RemoveContainer" containerID="5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9" Sep 10 16:08:11.719288 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:08:11.719273 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9\": container with ID starting with 5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9 not found: ID does not exist" containerID="5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9" Sep 10 16:08:11.719373 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.719311 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9"} err="failed to get container status \"5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9\": rpc error: code = NotFound desc = could not find container \"5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9\": container with ID starting with 5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9 not found: ID does not exist" Sep 10 16:08:11.719373 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.719332 2541 scope.go:117] "RemoveContainer" containerID="d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04" Sep 10 16:08:11.719587 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.719569 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04"} err="failed to get container status \"d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04\": rpc error: code = NotFound desc = could not find container \"d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04\": container with ID starting with d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04 not found: ID does not exist" Sep 10 16:08:11.719624 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.719589 2541 scope.go:117] "RemoveContainer" containerID="27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c" Sep 10 16:08:11.719798 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.719781 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c"} err="failed to get container status \"27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c\": rpc error: code = NotFound desc = could not find container \"27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c\": container with ID starting with 27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c not found: ID does not exist" Sep 10 16:08:11.719840 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.719798 2541 scope.go:117] "RemoveContainer" containerID="84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548" Sep 10 16:08:11.720011 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.719993 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548"} err="failed to get container status \"84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548\": rpc error: code = NotFound desc = could not find container \"84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548\": container with ID starting with 84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548 not found: ID does not exist" Sep 10 16:08:11.720079 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.720012 2541 scope.go:117] "RemoveContainer" containerID="195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929" Sep 10 16:08:11.720219 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.720204 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929"} err="failed to get container status \"195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929\": rpc error: code = NotFound desc = could not find container \"195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929\": container with ID starting with 195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929 not found: ID does not exist" Sep 10 16:08:11.720265 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.720220 2541 scope.go:117] "RemoveContainer" containerID="5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58" Sep 10 16:08:11.720454 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.720434 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58"} err="failed to get container status \"5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58\": rpc error: code = NotFound desc = could not find container \"5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58\": container with ID starting with 5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58 not found: ID does not exist" Sep 10 16:08:11.720500 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.720454 2541 scope.go:117] "RemoveContainer" containerID="71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3" Sep 10 16:08:11.720632 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.720613 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3"} err="failed to get container status \"71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3\": rpc error: code = NotFound desc = could not find container \"71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3\": container with ID starting with 71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3 not found: ID does not exist" Sep 10 16:08:11.720675 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.720634 2541 scope.go:117] "RemoveContainer" containerID="5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9" Sep 10 16:08:11.720835 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.720818 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9"} err="failed to get container status \"5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9\": rpc error: code = NotFound desc = could not find container \"5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9\": container with ID starting with 5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9 not found: ID does not exist" Sep 10 16:08:11.720947 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.720837 2541 scope.go:117] "RemoveContainer" containerID="d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04" Sep 10 16:08:11.721034 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.721015 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04"} err="failed to get container status \"d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04\": rpc error: code = NotFound desc = could not find container \"d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04\": container with ID starting with d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04 not found: ID does not exist" Sep 10 16:08:11.721077 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.721036 2541 scope.go:117] "RemoveContainer" containerID="27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c" Sep 10 16:08:11.721247 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.721228 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c"} err="failed to get container status \"27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c\": rpc error: code = NotFound desc = could not find container \"27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c\": container with ID starting with 27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c not found: ID does not exist" Sep 10 16:08:11.721548 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.721248 2541 scope.go:117] "RemoveContainer" containerID="84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548" Sep 10 16:08:11.721548 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.721476 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548"} err="failed to get container status \"84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548\": rpc error: code = NotFound desc = could not find container \"84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548\": container with ID starting with 84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548 not found: ID does not exist" Sep 10 16:08:11.721548 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.721487 2541 scope.go:117] "RemoveContainer" containerID="195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929" Sep 10 16:08:11.721675 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.721661 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929"} err="failed to get container status \"195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929\": rpc error: code = NotFound desc = could not find container \"195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929\": container with ID starting with 195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929 not found: ID does not exist" Sep 10 16:08:11.721720 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.721675 2541 scope.go:117] "RemoveContainer" containerID="5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58" Sep 10 16:08:11.721891 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.721873 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58"} err="failed to get container status \"5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58\": rpc error: code = NotFound desc = could not find container \"5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58\": container with ID starting with 5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58 not found: ID does not exist" Sep 10 16:08:11.721935 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.721893 2541 scope.go:117] "RemoveContainer" containerID="71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3" Sep 10 16:08:11.722110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.722093 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3"} err="failed to get container status \"71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3\": rpc error: code = NotFound desc = could not find container \"71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3\": container with ID starting with 71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3 not found: ID does not exist" Sep 10 16:08:11.722172 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.722112 2541 scope.go:117] "RemoveContainer" containerID="5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9" Sep 10 16:08:11.722329 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.722312 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9"} err="failed to get container status \"5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9\": rpc error: code = NotFound desc = could not find container \"5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9\": container with ID starting with 5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9 not found: ID does not exist" Sep 10 16:08:11.722373 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.722329 2541 scope.go:117] "RemoveContainer" containerID="d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04" Sep 10 16:08:11.722528 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.722511 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04"} err="failed to get container status \"d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04\": rpc error: code = NotFound desc = could not find container \"d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04\": container with ID starting with d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04 not found: ID does not exist" Sep 10 16:08:11.722571 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.722530 2541 scope.go:117] "RemoveContainer" containerID="27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c" Sep 10 16:08:11.722724 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.722706 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c"} err="failed to get container status \"27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c\": rpc error: code = NotFound desc = could not find container \"27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c\": container with ID starting with 27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c not found: ID does not exist" Sep 10 16:08:11.722789 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.722726 2541 scope.go:117] "RemoveContainer" containerID="84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548" Sep 10 16:08:11.722954 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.722935 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548"} err="failed to get container status \"84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548\": rpc error: code = NotFound desc = could not find container \"84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548\": container with ID starting with 84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548 not found: ID does not exist" Sep 10 16:08:11.722954 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.722951 2541 scope.go:117] "RemoveContainer" containerID="195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929" Sep 10 16:08:11.723149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.723132 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929"} err="failed to get container status \"195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929\": rpc error: code = NotFound desc = could not find container \"195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929\": container with ID starting with 195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929 not found: ID does not exist" Sep 10 16:08:11.723193 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.723150 2541 scope.go:117] "RemoveContainer" containerID="5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58" Sep 10 16:08:11.723364 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.723349 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58"} err="failed to get container status \"5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58\": rpc error: code = NotFound desc = could not find container \"5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58\": container with ID starting with 5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58 not found: ID does not exist" Sep 10 16:08:11.723426 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.723364 2541 scope.go:117] "RemoveContainer" containerID="71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3" Sep 10 16:08:11.723573 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.723558 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3"} err="failed to get container status \"71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3\": rpc error: code = NotFound desc = could not find container \"71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3\": container with ID starting with 71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3 not found: ID does not exist" Sep 10 16:08:11.723573 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.723572 2541 scope.go:117] "RemoveContainer" containerID="5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9" Sep 10 16:08:11.723742 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.723727 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9"} err="failed to get container status \"5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9\": rpc error: code = NotFound desc = could not find container \"5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9\": container with ID starting with 5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9 not found: ID does not exist" Sep 10 16:08:11.723742 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.723741 2541 scope.go:117] "RemoveContainer" containerID="d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04" Sep 10 16:08:11.723934 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.723918 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04"} err="failed to get container status \"d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04\": rpc error: code = NotFound desc = could not find container \"d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04\": container with ID starting with d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04 not found: ID does not exist" Sep 10 16:08:11.723979 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.723935 2541 scope.go:117] "RemoveContainer" containerID="27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c" Sep 10 16:08:11.724122 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.724106 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c"} err="failed to get container status \"27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c\": rpc error: code = NotFound desc = could not find container \"27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c\": container with ID starting with 27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c not found: ID does not exist" Sep 10 16:08:11.724166 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.724124 2541 scope.go:117] "RemoveContainer" containerID="84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548" Sep 10 16:08:11.724355 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.724339 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548"} err="failed to get container status \"84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548\": rpc error: code = NotFound desc = could not find container \"84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548\": container with ID starting with 84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548 not found: ID does not exist" Sep 10 16:08:11.724417 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.724356 2541 scope.go:117] "RemoveContainer" containerID="195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929" Sep 10 16:08:11.724546 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.724531 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929"} err="failed to get container status \"195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929\": rpc error: code = NotFound desc = could not find container \"195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929\": container with ID starting with 195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929 not found: ID does not exist" Sep 10 16:08:11.724583 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.724546 2541 scope.go:117] "RemoveContainer" containerID="5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58" Sep 10 16:08:11.724737 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.724722 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58"} err="failed to get container status \"5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58\": rpc error: code = NotFound desc = could not find container \"5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58\": container with ID starting with 5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58 not found: ID does not exist" Sep 10 16:08:11.724780 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.724738 2541 scope.go:117] "RemoveContainer" containerID="71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3" Sep 10 16:08:11.724910 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.724896 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3"} err="failed to get container status \"71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3\": rpc error: code = NotFound desc = could not find container \"71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3\": container with ID starting with 71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3 not found: ID does not exist" Sep 10 16:08:11.724963 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.724911 2541 scope.go:117] "RemoveContainer" containerID="5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9" Sep 10 16:08:11.725106 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.725084 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9"} err="failed to get container status \"5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9\": rpc error: code = NotFound desc = could not find container \"5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9\": container with ID starting with 5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9 not found: ID does not exist" Sep 10 16:08:11.725106 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.725099 2541 scope.go:117] "RemoveContainer" containerID="d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04" Sep 10 16:08:11.725268 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.725254 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04"} err="failed to get container status \"d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04\": rpc error: code = NotFound desc = could not find container \"d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04\": container with ID starting with d6998dd434b97b190bd91ec082db818f25f74a6a5218cdd863bed34cc2403b04 not found: ID does not exist" Sep 10 16:08:11.725333 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.725269 2541 scope.go:117] "RemoveContainer" containerID="27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c" Sep 10 16:08:11.725510 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.725492 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c"} err="failed to get container status \"27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c\": rpc error: code = NotFound desc = could not find container \"27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c\": container with ID starting with 27c3150ac0abbb8caea0d48d2353c35ad87beeb81b6b9b48c0bbec2f8bb7e23c not found: ID does not exist" Sep 10 16:08:11.725574 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.725514 2541 scope.go:117] "RemoveContainer" containerID="84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548" Sep 10 16:08:11.725748 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.725730 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548"} err="failed to get container status \"84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548\": rpc error: code = NotFound desc = could not find container \"84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548\": container with ID starting with 84ab68eaa4bb599fad549e9b0b3a5af6aadae492cbfec3e1644fe230815cd548 not found: ID does not exist" Sep 10 16:08:11.725799 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.725748 2541 scope.go:117] "RemoveContainer" containerID="195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929" Sep 10 16:08:11.725956 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.725937 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929"} err="failed to get container status \"195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929\": rpc error: code = NotFound desc = could not find container \"195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929\": container with ID starting with 195fcaad9fc81b4c84de8a0143868d63a4746f45b69925f72d2dfca39ad58929 not found: ID does not exist" Sep 10 16:08:11.726002 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.725957 2541 scope.go:117] "RemoveContainer" containerID="5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58" Sep 10 16:08:11.726194 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.726175 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58"} err="failed to get container status \"5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58\": rpc error: code = NotFound desc = could not find container \"5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58\": container with ID starting with 5eacc1559b813f91cb7124837b8430a72ae1eabafb495d7eae5e4d010810ce58 not found: ID does not exist" Sep 10 16:08:11.726239 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.726196 2541 scope.go:117] "RemoveContainer" containerID="71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3" Sep 10 16:08:11.726509 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.726453 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3"} err="failed to get container status \"71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3\": rpc error: code = NotFound desc = could not find container \"71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3\": container with ID starting with 71390609abf0d2f20f4ba7824da01b11cc176c85eb1579503c9aa5ccd7b927d3 not found: ID does not exist" Sep 10 16:08:11.726509 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.726472 2541 scope.go:117] "RemoveContainer" containerID="5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9" Sep 10 16:08:11.726735 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.726714 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9"} err="failed to get container status \"5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9\": rpc error: code = NotFound desc = could not find container \"5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9\": container with ID starting with 5d6e7dea0303afd3ab408b5078d5fb0073934b97899ba9957bc4c7b40ae264c9 not found: ID does not exist" Sep 10 16:08:11.738649 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.738625 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 16:08:11.738941 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.738927 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="config-reloader" Sep 10 16:08:11.738988 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.738943 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="config-reloader" Sep 10 16:08:11.738988 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.738952 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="kube-rbac-proxy-web" Sep 10 16:08:11.738988 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.738960 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="kube-rbac-proxy-web" Sep 10 16:08:11.738988 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.738969 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad" containerName="registry" Sep 10 16:08:11.738988 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.738978 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad" containerName="registry" Sep 10 16:08:11.739134 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.738989 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="alertmanager" Sep 10 16:08:11.739134 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.738997 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="alertmanager" Sep 10 16:08:11.739134 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.739007 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="kube-rbac-proxy-metric" Sep 10 16:08:11.739134 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.739012 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="kube-rbac-proxy-metric" Sep 10 16:08:11.739134 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.739020 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="init-config-reloader" Sep 10 16:08:11.739134 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.739027 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="init-config-reloader" Sep 10 16:08:11.739134 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.739034 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59656fa7-1ec7-40d1-bbbc-73927c3f4ff6" containerName="console" Sep 10 16:08:11.739134 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.739039 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="59656fa7-1ec7-40d1-bbbc-73927c3f4ff6" containerName="console" Sep 10 16:08:11.739134 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.739045 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1abf66ea-0b80-47d4-b51e-670b7618c703" containerName="console" Sep 10 16:08:11.739134 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.739050 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="1abf66ea-0b80-47d4-b51e-670b7618c703" containerName="console" Sep 10 16:08:11.739134 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.739057 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="kube-rbac-proxy" Sep 10 16:08:11.739134 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.739061 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="kube-rbac-proxy" Sep 10 16:08:11.739134 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.739074 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="prom-label-proxy" Sep 10 16:08:11.739134 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.739083 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="prom-label-proxy" Sep 10 16:08:11.739134 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.739125 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="alertmanager" Sep 10 16:08:11.739134 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.739133 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="kube-rbac-proxy-web" Sep 10 16:08:11.739648 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.739140 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="13f79cf2-a9e2-45ac-8f66-9a5e35cb7fad" containerName="registry" Sep 10 16:08:11.739648 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.739147 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="kube-rbac-proxy-metric" Sep 10 16:08:11.739648 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.739154 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="59656fa7-1ec7-40d1-bbbc-73927c3f4ff6" containerName="console" Sep 10 16:08:11.739648 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.739162 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="config-reloader" Sep 10 16:08:11.739648 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.739171 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="kube-rbac-proxy" Sep 10 16:08:11.739648 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.739175 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" containerName="prom-label-proxy" Sep 10 16:08:11.739648 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.739180 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="1abf66ea-0b80-47d4-b51e-670b7618c703" containerName="console" Sep 10 16:08:11.745115 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.745096 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:11.747377 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.747356 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 16:08:11.747479 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.747353 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 16:08:11.747540 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.747397 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 16:08:11.747651 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.747625 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 16:08:11.747803 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.747714 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 16:08:11.747803 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.747714 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 16:08:11.747803 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.747749 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 16:08:11.747969 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.747814 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 16:08:11.747969 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.747957 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6p9l6\"" Sep 10 16:08:11.753024 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.753002 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 16:08:11.758682 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.758662 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 16:08:11.928389 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.928260 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f035c949-761e-4516-9b3e-1d48e505b056-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:11.928389 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.928341 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bxc7\" (UniqueName: \"kubernetes.io/projected/f035c949-761e-4516-9b3e-1d48e505b056-kube-api-access-9bxc7\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:11.928389 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.928375 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f035c949-761e-4516-9b3e-1d48e505b056-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:11.928640 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.928399 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f035c949-761e-4516-9b3e-1d48e505b056-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:11.928640 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.928421 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f035c949-761e-4516-9b3e-1d48e505b056-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:11.928640 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.928466 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f035c949-761e-4516-9b3e-1d48e505b056-config-out\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:11.928640 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.928508 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f035c949-761e-4516-9b3e-1d48e505b056-web-config\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:11.928640 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.928550 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f035c949-761e-4516-9b3e-1d48e505b056-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:11.928640 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.928591 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f035c949-761e-4516-9b3e-1d48e505b056-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:11.928640 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.928614 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f035c949-761e-4516-9b3e-1d48e505b056-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:11.928886 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.928671 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f035c949-761e-4516-9b3e-1d48e505b056-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:11.928886 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.928697 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f035c949-761e-4516-9b3e-1d48e505b056-config-volume\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:11.928886 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:11.928714 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f035c949-761e-4516-9b3e-1d48e505b056-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.029272 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.029235 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f035c949-761e-4516-9b3e-1d48e505b056-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.029272 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.029278 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f035c949-761e-4516-9b3e-1d48e505b056-config-volume\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.029528 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.029325 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f035c949-761e-4516-9b3e-1d48e505b056-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.029528 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.029346 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f035c949-761e-4516-9b3e-1d48e505b056-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.029528 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.029363 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bxc7\" (UniqueName: \"kubernetes.io/projected/f035c949-761e-4516-9b3e-1d48e505b056-kube-api-access-9bxc7\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.029528 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.029391 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f035c949-761e-4516-9b3e-1d48e505b056-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.029528 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.029407 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f035c949-761e-4516-9b3e-1d48e505b056-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.029528 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.029422 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f035c949-761e-4516-9b3e-1d48e505b056-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.029528 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.029470 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f035c949-761e-4516-9b3e-1d48e505b056-config-out\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.029528 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.029494 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f035c949-761e-4516-9b3e-1d48e505b056-web-config\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.029528 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.029522 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f035c949-761e-4516-9b3e-1d48e505b056-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.030007 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.029558 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f035c949-761e-4516-9b3e-1d48e505b056-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.030007 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.029583 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f035c949-761e-4516-9b3e-1d48e505b056-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.030155 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.030131 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f035c949-761e-4516-9b3e-1d48e505b056-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.030247 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.030223 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f035c949-761e-4516-9b3e-1d48e505b056-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.030324 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.030278 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f035c949-761e-4516-9b3e-1d48e505b056-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.032969 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.032939 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f035c949-761e-4516-9b3e-1d48e505b056-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.033152 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.033128 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f035c949-761e-4516-9b3e-1d48e505b056-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.033347 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.033327 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f035c949-761e-4516-9b3e-1d48e505b056-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.033347 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.033343 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f035c949-761e-4516-9b3e-1d48e505b056-config-volume\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.033581 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.033544 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f035c949-761e-4516-9b3e-1d48e505b056-config-out\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.033671 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.033608 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f035c949-761e-4516-9b3e-1d48e505b056-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.033901 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.033885 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f035c949-761e-4516-9b3e-1d48e505b056-web-config\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.033962 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.033930 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f035c949-761e-4516-9b3e-1d48e505b056-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.034836 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.034817 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f035c949-761e-4516-9b3e-1d48e505b056-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.040103 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.040082 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bxc7\" (UniqueName: \"kubernetes.io/projected/f035c949-761e-4516-9b3e-1d48e505b056-kube-api-access-9bxc7\") pod \"alertmanager-main-0\" (UID: \"f035c949-761e-4516-9b3e-1d48e505b056\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.054972 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.054954 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:08:12.344599 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.344565 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 16:08:12.346687 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:08:12.346664 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf035c949_761e_4516_9b3e_1d48e505b056.slice/crio-47247c0f696710ab051288a3929d1a57844f6b10630a728ce7bebadfc1c71fe1 WatchSource:0}: Error finding container 47247c0f696710ab051288a3929d1a57844f6b10630a728ce7bebadfc1c71fe1: Status 404 returned error can't find the container with id 47247c0f696710ab051288a3929d1a57844f6b10630a728ce7bebadfc1c71fe1 Sep 10 16:08:12.673087 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.673048 2541 generic.go:356] "Generic (PLEG): container finished" podID="f035c949-761e-4516-9b3e-1d48e505b056" containerID="fad42f4132f5a97f9cd8e1435a454835dfc5b7c7f2d6553fa93db972ddc3ea1f" exitCode=0 Sep 10 16:08:12.673275 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.673132 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f035c949-761e-4516-9b3e-1d48e505b056","Type":"ContainerDied","Data":"fad42f4132f5a97f9cd8e1435a454835dfc5b7c7f2d6553fa93db972ddc3ea1f"} Sep 10 16:08:12.673275 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.673214 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f035c949-761e-4516-9b3e-1d48e505b056","Type":"ContainerStarted","Data":"47247c0f696710ab051288a3929d1a57844f6b10630a728ce7bebadfc1c71fe1"} Sep 10 16:08:12.978675 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:12.978653 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e2088c6-e4c2-485a-b3c7-094b1e42b402" path="/var/lib/kubelet/pods/1e2088c6-e4c2-485a-b3c7-094b1e42b402/volumes" Sep 10 16:08:13.464214 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:13.464185 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:13.532209 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:13.532175 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:13.681887 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:13.681852 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f035c949-761e-4516-9b3e-1d48e505b056","Type":"ContainerStarted","Data":"b878ef73c5bba634583e21bf0409290d2a1104992f05af54a95ee51ba2b32c8d"} Sep 10 16:08:13.682074 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:13.681930 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f035c949-761e-4516-9b3e-1d48e505b056","Type":"ContainerStarted","Data":"fd3ad43eaace791e5fe17029d6d3ca88301ef45800334b6daa3e77c7a97215d8"} Sep 10 16:08:13.682074 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:13.681978 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f035c949-761e-4516-9b3e-1d48e505b056","Type":"ContainerStarted","Data":"f0b96e16cb141b18ec10f8cf5c05f8cd6f742e11b9936c759435ffbd0a447efc"} Sep 10 16:08:13.682074 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:13.682016 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f035c949-761e-4516-9b3e-1d48e505b056","Type":"ContainerStarted","Data":"c9cb04de0f5d6ae8fff79b66d1b82e3965e725c1be1ec71e084316ab6f5c394a"} Sep 10 16:08:13.682074 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:13.682053 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f035c949-761e-4516-9b3e-1d48e505b056","Type":"ContainerStarted","Data":"c139222c4673306f8b51c47c4471026c94fb3f83b65393fee9771d16b3190855"} Sep 10 16:08:13.682218 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:13.682089 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f035c949-761e-4516-9b3e-1d48e505b056","Type":"ContainerStarted","Data":"0c33196f661b0eff3c50507c6474174158e7e55c91f32e4432e99791011c3939"} Sep 10 16:08:13.721437 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:13.721325 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.705547176 podStartE2EDuration="2.721293837s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.746661ms" isStatefulPod=false podCreationTimestamp="2026-09-10 16:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:08:13.720103525 +0000 UTC m=+393.243348745" watchObservedRunningTime="2026-09-10 16:08:13.721293837 +0000 UTC m=+393.244539035" Sep 10 16:08:16.117840 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:16.117802 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:37.271525 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:08:37.271486 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:26.089036 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:26.089004 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:26.170698 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:26.170664 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:35.775647 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.775615 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6787b8b9fd-ghlgz"] Sep 10 16:09:35.778872 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.778856 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:35.798321 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.797705 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6787b8b9fd-ghlgz"] Sep 10 16:09:35.861176 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.861099 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:35.885021 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.884978 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/312fe479-86ab-456a-89f2-7a5480a95bae-console-oauth-config\") pod \"console-6787b8b9fd-ghlgz\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:35.885021 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.885020 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-trusted-ca-bundle\") pod \"console-6787b8b9fd-ghlgz\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:35.885228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.885062 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-console-config\") pod \"console-6787b8b9fd-ghlgz\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:35.885228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.885090 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/312fe479-86ab-456a-89f2-7a5480a95bae-console-serving-cert\") pod \"console-6787b8b9fd-ghlgz\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:35.885228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.885118 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-service-ca\") pod \"console-6787b8b9fd-ghlgz\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:35.885228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.885172 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-oauth-serving-cert\") pod \"console-6787b8b9fd-ghlgz\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:35.885228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.885227 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fjq4\" (UniqueName: \"kubernetes.io/projected/312fe479-86ab-456a-89f2-7a5480a95bae-kube-api-access-2fjq4\") pod \"console-6787b8b9fd-ghlgz\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:35.938874 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.938844 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:35.985827 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.985795 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-service-ca\") pod \"console-6787b8b9fd-ghlgz\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:35.985950 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.985831 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-oauth-serving-cert\") pod \"console-6787b8b9fd-ghlgz\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:35.985950 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.985860 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fjq4\" (UniqueName: \"kubernetes.io/projected/312fe479-86ab-456a-89f2-7a5480a95bae-kube-api-access-2fjq4\") pod \"console-6787b8b9fd-ghlgz\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:35.985950 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.985885 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/312fe479-86ab-456a-89f2-7a5480a95bae-console-oauth-config\") pod \"console-6787b8b9fd-ghlgz\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:35.985950 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.985904 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-trusted-ca-bundle\") pod \"console-6787b8b9fd-ghlgz\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:35.985950 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.985941 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-console-config\") pod \"console-6787b8b9fd-ghlgz\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:35.986206 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.985971 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/312fe479-86ab-456a-89f2-7a5480a95bae-console-serving-cert\") pod \"console-6787b8b9fd-ghlgz\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:35.986685 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.986659 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-service-ca\") pod \"console-6787b8b9fd-ghlgz\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:35.986795 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.986659 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-oauth-serving-cert\") pod \"console-6787b8b9fd-ghlgz\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:35.986906 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.986883 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-trusted-ca-bundle\") pod \"console-6787b8b9fd-ghlgz\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:35.986943 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.986886 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-console-config\") pod \"console-6787b8b9fd-ghlgz\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:35.988460 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.988440 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/312fe479-86ab-456a-89f2-7a5480a95bae-console-oauth-config\") pod \"console-6787b8b9fd-ghlgz\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:35.988578 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.988560 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/312fe479-86ab-456a-89f2-7a5480a95bae-console-serving-cert\") pod \"console-6787b8b9fd-ghlgz\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:35.995663 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:35.995639 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fjq4\" (UniqueName: \"kubernetes.io/projected/312fe479-86ab-456a-89f2-7a5480a95bae-kube-api-access-2fjq4\") pod \"console-6787b8b9fd-ghlgz\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:36.089513 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:36.089431 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:36.363484 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:36.363406 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6787b8b9fd-ghlgz"] Sep 10 16:09:36.366689 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:09:36.366665 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312fe479_86ab_456a_89f2_7a5480a95bae.slice/crio-0ce32e2173c8608cda4bbc470bf902bd592048e853bc88b9c9f3310ff8639891 WatchSource:0}: Error finding container 0ce32e2173c8608cda4bbc470bf902bd592048e853bc88b9c9f3310ff8639891: Status 404 returned error can't find the container with id 0ce32e2173c8608cda4bbc470bf902bd592048e853bc88b9c9f3310ff8639891 Sep 10 16:09:37.004416 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:37.004386 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6787b8b9fd-ghlgz" event={"ID":"312fe479-86ab-456a-89f2-7a5480a95bae","Type":"ContainerStarted","Data":"9d9ce547dacffbddba9320afb0d8c5b33a2bec8e54c9c3e3750e1dec6b2dca38"} Sep 10 16:09:37.004751 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:37.004432 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6787b8b9fd-ghlgz" event={"ID":"312fe479-86ab-456a-89f2-7a5480a95bae","Type":"ContainerStarted","Data":"0ce32e2173c8608cda4bbc470bf902bd592048e853bc88b9c9f3310ff8639891"} Sep 10 16:09:37.027111 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:37.027069 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6787b8b9fd-ghlgz" podStartSLOduration=2.027050566 podStartE2EDuration="2.027050566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:09:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:09:37.027030673 +0000 UTC m=+476.550275871" watchObservedRunningTime="2026-09-10 16:09:37.027050566 +0000 UTC m=+476.550295765" Sep 10 16:09:38.086258 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:38.086223 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:38.169694 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:38.169662 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:46.089673 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:46.089637 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:46.090109 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:46.089691 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:46.096322 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:46.095592 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:47.048104 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:47.048075 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:09:47.122016 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:47.121975 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fbdfdfbbf-zfxjn"] Sep 10 16:09:47.289929 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:47.289897 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:47.394957 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:47.394925 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:53.028197 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:53.028165 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:53.132407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:53.132370 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:53.406933 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:09:53.406902 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:12.143490 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.143426 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-fbdfdfbbf-zfxjn" podUID="8d9d5450-eccc-4220-9430-3dc0b43a7e65" containerName="console" containerID="cri-o://8c20761464a1e55673ba4be3cfcbc4867cc0f61a414056bf5b78221a6d9e28fb" gracePeriod=15 Sep 10 16:10:12.378681 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.378658 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:10:12.490921 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.490838 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hj5nn\" (UniqueName: \"kubernetes.io/projected/8d9d5450-eccc-4220-9430-3dc0b43a7e65-kube-api-access-hj5nn\") pod \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " Sep 10 16:10:12.490921 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.490874 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-oauth-serving-cert\") pod \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " Sep 10 16:10:12.490921 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.490915 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d9d5450-eccc-4220-9430-3dc0b43a7e65-console-serving-cert\") pod \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " Sep 10 16:10:12.491185 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.490944 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-trusted-ca-bundle\") pod \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " Sep 10 16:10:12.491185 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.490961 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-service-ca\") pod \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " Sep 10 16:10:12.491185 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.490978 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-console-config\") pod \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " Sep 10 16:10:12.491185 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.491020 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d9d5450-eccc-4220-9430-3dc0b43a7e65-console-oauth-config\") pod \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\" (UID: \"8d9d5450-eccc-4220-9430-3dc0b43a7e65\") " Sep 10 16:10:12.491443 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.491412 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-service-ca" (OuterVolumeSpecName: "service-ca") pod "8d9d5450-eccc-4220-9430-3dc0b43a7e65" (UID: "8d9d5450-eccc-4220-9430-3dc0b43a7e65"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:10:12.491504 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.491430 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8d9d5450-eccc-4220-9430-3dc0b43a7e65" (UID: "8d9d5450-eccc-4220-9430-3dc0b43a7e65"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:10:12.491504 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.491439 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-console-config" (OuterVolumeSpecName: "console-config") pod "8d9d5450-eccc-4220-9430-3dc0b43a7e65" (UID: "8d9d5450-eccc-4220-9430-3dc0b43a7e65"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:10:12.491504 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.491474 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8d9d5450-eccc-4220-9430-3dc0b43a7e65" (UID: "8d9d5450-eccc-4220-9430-3dc0b43a7e65"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:10:12.493171 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.493144 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9d5450-eccc-4220-9430-3dc0b43a7e65-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8d9d5450-eccc-4220-9430-3dc0b43a7e65" (UID: "8d9d5450-eccc-4220-9430-3dc0b43a7e65"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:10:12.493294 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.493174 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9d5450-eccc-4220-9430-3dc0b43a7e65-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8d9d5450-eccc-4220-9430-3dc0b43a7e65" (UID: "8d9d5450-eccc-4220-9430-3dc0b43a7e65"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:10:12.493294 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.493250 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9d5450-eccc-4220-9430-3dc0b43a7e65-kube-api-access-hj5nn" (OuterVolumeSpecName: "kube-api-access-hj5nn") pod "8d9d5450-eccc-4220-9430-3dc0b43a7e65" (UID: "8d9d5450-eccc-4220-9430-3dc0b43a7e65"). InnerVolumeSpecName "kube-api-access-hj5nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:10:12.592129 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.592094 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj5nn\" (UniqueName: \"kubernetes.io/projected/8d9d5450-eccc-4220-9430-3dc0b43a7e65-kube-api-access-hj5nn\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:10:12.592129 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.592119 2541 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-oauth-serving-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:10:12.592129 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.592129 2541 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d9d5450-eccc-4220-9430-3dc0b43a7e65-console-serving-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:10:12.592370 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.592139 2541 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-trusted-ca-bundle\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:10:12.592370 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.592148 2541 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:10:12.592370 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.592156 2541 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d9d5450-eccc-4220-9430-3dc0b43a7e65-console-config\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:10:12.592370 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:12.592165 2541 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d9d5450-eccc-4220-9430-3dc0b43a7e65-console-oauth-config\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:10:13.142964 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:13.142925 2541 generic.go:356] "Generic (PLEG): container finished" podID="8d9d5450-eccc-4220-9430-3dc0b43a7e65" containerID="8c20761464a1e55673ba4be3cfcbc4867cc0f61a414056bf5b78221a6d9e28fb" exitCode=0 Sep 10 16:10:13.143142 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:13.142976 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbdfdfbbf-zfxjn" Sep 10 16:10:13.143142 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:13.143006 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbdfdfbbf-zfxjn" event={"ID":"8d9d5450-eccc-4220-9430-3dc0b43a7e65","Type":"ContainerDied","Data":"8c20761464a1e55673ba4be3cfcbc4867cc0f61a414056bf5b78221a6d9e28fb"} Sep 10 16:10:13.143142 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:13.143060 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbdfdfbbf-zfxjn" event={"ID":"8d9d5450-eccc-4220-9430-3dc0b43a7e65","Type":"ContainerDied","Data":"cb523e353f52d8d518e418ff425c7d0540902df7d21fea924c88754cfec36f16"} Sep 10 16:10:13.143142 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:13.143081 2541 scope.go:117] "RemoveContainer" containerID="8c20761464a1e55673ba4be3cfcbc4867cc0f61a414056bf5b78221a6d9e28fb" Sep 10 16:10:13.151343 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:13.151291 2541 scope.go:117] "RemoveContainer" containerID="8c20761464a1e55673ba4be3cfcbc4867cc0f61a414056bf5b78221a6d9e28fb" Sep 10 16:10:13.151663 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:10:13.151645 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c20761464a1e55673ba4be3cfcbc4867cc0f61a414056bf5b78221a6d9e28fb\": container with ID starting with 8c20761464a1e55673ba4be3cfcbc4867cc0f61a414056bf5b78221a6d9e28fb not found: ID does not exist" containerID="8c20761464a1e55673ba4be3cfcbc4867cc0f61a414056bf5b78221a6d9e28fb" Sep 10 16:10:13.151708 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:13.151675 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c20761464a1e55673ba4be3cfcbc4867cc0f61a414056bf5b78221a6d9e28fb"} err="failed to get container status \"8c20761464a1e55673ba4be3cfcbc4867cc0f61a414056bf5b78221a6d9e28fb\": rpc error: code = NotFound desc = could not find container \"8c20761464a1e55673ba4be3cfcbc4867cc0f61a414056bf5b78221a6d9e28fb\": container with ID starting with 8c20761464a1e55673ba4be3cfcbc4867cc0f61a414056bf5b78221a6d9e28fb not found: ID does not exist" Sep 10 16:10:13.163086 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:13.163062 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fbdfdfbbf-zfxjn"] Sep 10 16:10:13.172442 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:13.172404 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fbdfdfbbf-zfxjn"] Sep 10 16:10:14.978472 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:14.978434 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d9d5450-eccc-4220-9430-3dc0b43a7e65" path="/var/lib/kubelet/pods/8d9d5450-eccc-4220-9430-3dc0b43a7e65/volumes" Sep 10 16:10:38.241641 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:38.241567 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:39.464739 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:39.464708 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd"] Sep 10 16:10:39.465104 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:39.464991 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9d5450-eccc-4220-9430-3dc0b43a7e65" containerName="console" Sep 10 16:10:39.465104 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:39.465002 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9d5450-eccc-4220-9430-3dc0b43a7e65" containerName="console" Sep 10 16:10:39.465104 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:39.465085 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d9d5450-eccc-4220-9430-3dc0b43a7e65" containerName="console" Sep 10 16:10:39.468065 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:39.468050 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd" Sep 10 16:10:39.473385 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:10:39.473350 2541 status_manager.go:1018] "Failed to get status for pod" err="pods \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-marketplace\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd" Sep 10 16:10:39.473385 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:10:39.473373 2541 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-marketplace\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Sep 10 16:10:39.473603 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:10:39.473375 2541 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-marketplace\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Sep 10 16:10:39.473603 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:10:39.473419 2541 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-hcb7s\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-marketplace\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hcb7s\"" type="*v1.Secret" Sep 10 16:10:39.492009 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:39.491973 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd"] Sep 10 16:10:39.495406 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:39.495375 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd\" (UID: \"ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd" Sep 10 16:10:39.495523 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:39.495470 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckd5n\" (UniqueName: \"kubernetes.io/projected/ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153-kube-api-access-ckd5n\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd\" (UID: \"ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd" Sep 10 16:10:39.495573 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:39.495540 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd\" (UID: \"ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd" Sep 10 16:10:39.596789 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:39.596754 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd\" (UID: \"ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd" Sep 10 16:10:39.596953 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:39.596799 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd\" (UID: \"ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd" Sep 10 16:10:39.596953 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:39.596856 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckd5n\" (UniqueName: \"kubernetes.io/projected/ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153-kube-api-access-ckd5n\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd\" (UID: \"ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd" Sep 10 16:10:39.597217 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:39.597194 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd\" (UID: \"ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd" Sep 10 16:10:39.597290 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:39.597214 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd\" (UID: \"ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd" Sep 10 16:10:39.924549 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:39.924511 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:40.355523 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:40.355441 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 16:10:40.466792 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:40.466767 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hcb7s\"" Sep 10 16:10:40.605748 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:10:40.605654 2541 projected.go:291] Couldn't get configMap openshift-marketplace/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition Sep 10 16:10:40.605748 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:10:40.605708 2541 projected.go:196] Error preparing data for projected volume kube-api-access-ckd5n for pod openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd: failed to sync configmap cache: timed out waiting for the condition Sep 10 16:10:40.605947 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:10:40.605784 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153-kube-api-access-ckd5n podName:ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153 nodeName:}" failed. No retries permitted until 2026-09-10 16:10:41.105766887 +0000 UTC m=+540.629012063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ckd5n" (UniqueName: "kubernetes.io/projected/ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153-kube-api-access-ckd5n") pod "685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd" (UID: "ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153") : failed to sync configmap cache: timed out waiting for the condition Sep 10 16:10:40.814412 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:40.814384 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 16:10:41.110231 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:41.110201 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckd5n\" (UniqueName: \"kubernetes.io/projected/ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153-kube-api-access-ckd5n\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd\" (UID: \"ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd" Sep 10 16:10:41.113083 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:41.113053 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckd5n\" (UniqueName: \"kubernetes.io/projected/ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153-kube-api-access-ckd5n\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd\" (UID: \"ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd" Sep 10 16:10:41.278958 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:41.278876 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hcb7s\"" Sep 10 16:10:41.287136 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:41.287110 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd" Sep 10 16:10:41.562210 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:41.562183 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd"] Sep 10 16:10:41.564903 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:10:41.564878 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbb7a36_6fb4_4bbb_b80b_9b43d4cfe153.slice/crio-6993d8d6bef2e8062915267adfb1b240632b4eb73de1062430d0537cb22845aa WatchSource:0}: Error finding container 6993d8d6bef2e8062915267adfb1b240632b4eb73de1062430d0537cb22845aa: Status 404 returned error can't find the container with id 6993d8d6bef2e8062915267adfb1b240632b4eb73de1062430d0537cb22845aa Sep 10 16:10:41.567268 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:41.567253 2541 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 16:10:42.258528 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:42.258490 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd" event={"ID":"ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153","Type":"ContainerStarted","Data":"6993d8d6bef2e8062915267adfb1b240632b4eb73de1062430d0537cb22845aa"} Sep 10 16:10:48.287689 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:48.287651 2541 generic.go:356] "Generic (PLEG): container finished" podID="ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153" containerID="e7493c7919503d76ec66f19ed12d1a700f8905074b3515a50825403db8968ee4" exitCode=0 Sep 10 16:10:48.288145 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:48.287743 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd" event={"ID":"ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153","Type":"ContainerDied","Data":"e7493c7919503d76ec66f19ed12d1a700f8905074b3515a50825403db8968ee4"} Sep 10 16:10:51.303537 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:51.303499 2541 generic.go:356] "Generic (PLEG): container finished" podID="ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153" containerID="8276525328eaf1d2b8ab724deb3e7d2f230f9a9533af702876feab09a83763e5" exitCode=0 Sep 10 16:10:51.303924 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:51.303585 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd" event={"ID":"ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153","Type":"ContainerDied","Data":"8276525328eaf1d2b8ab724deb3e7d2f230f9a9533af702876feab09a83763e5"} Sep 10 16:10:58.342362 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:58.342270 2541 generic.go:356] "Generic (PLEG): container finished" podID="ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153" containerID="40cf10ee2613684eb899288f89232c6d4f98b6106fed454d9fc061ef1f380c48" exitCode=0 Sep 10 16:10:58.342739 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:58.342357 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd" event={"ID":"ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153","Type":"ContainerDied","Data":"40cf10ee2613684eb899288f89232c6d4f98b6106fed454d9fc061ef1f380c48"} Sep 10 16:10:59.514271 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:59.514239 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd" Sep 10 16:10:59.570581 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:59.570552 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckd5n\" (UniqueName: \"kubernetes.io/projected/ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153-kube-api-access-ckd5n\") pod \"ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153\" (UID: \"ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153\") " Sep 10 16:10:59.570734 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:59.570601 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153-util\") pod \"ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153\" (UID: \"ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153\") " Sep 10 16:10:59.570734 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:59.570673 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153-bundle\") pod \"ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153\" (UID: \"ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153\") " Sep 10 16:10:59.571449 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:59.571419 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153-bundle" (OuterVolumeSpecName: "bundle") pod "ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153" (UID: "ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:10:59.572845 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:59.572815 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153-kube-api-access-ckd5n" (OuterVolumeSpecName: "kube-api-access-ckd5n") pod "ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153" (UID: "ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153"). InnerVolumeSpecName "kube-api-access-ckd5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:10:59.575763 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:59.575743 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153-util" (OuterVolumeSpecName: "util") pod "ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153" (UID: "ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:10:59.671984 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:59.671944 2541 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153-util\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:10:59.671984 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:59.671976 2541 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153-bundle\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:10:59.671984 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:10:59.671990 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckd5n\" (UniqueName: \"kubernetes.io/projected/ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153-kube-api-access-ckd5n\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:11:00.351643 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:00.351605 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd" event={"ID":"ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153","Type":"ContainerDied","Data":"6993d8d6bef2e8062915267adfb1b240632b4eb73de1062430d0537cb22845aa"} Sep 10 16:11:00.351818 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:00.351651 2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6993d8d6bef2e8062915267adfb1b240632b4eb73de1062430d0537cb22845aa" Sep 10 16:11:00.351818 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:00.351616 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rwwxd" Sep 10 16:11:05.573418 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:05.573382 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:06.438700 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:06.438660 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:07.576048 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:07.576008 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mc2td"] Sep 10 16:11:07.576529 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:07.576440 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153" containerName="pull" Sep 10 16:11:07.576529 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:07.576457 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153" containerName="pull" Sep 10 16:11:07.576529 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:07.576487 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153" containerName="extract" Sep 10 16:11:07.576529 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:07.576495 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153" containerName="extract" Sep 10 16:11:07.576529 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:07.576504 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153" containerName="util" Sep 10 16:11:07.576529 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:07.576511 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153" containerName="util" Sep 10 16:11:07.576820 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:07.576583 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccbb7a36-6fb4-4bbb-b80b-9b43d4cfe153" containerName="extract" Sep 10 16:11:07.579598 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:07.579576 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mc2td" Sep 10 16:11:07.583088 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:07.583068 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 10 16:11:07.583549 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:07.583536 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 10 16:11:07.586266 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:07.586247 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-p88bh\"" Sep 10 16:11:07.595372 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:07.595352 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mc2td"] Sep 10 16:11:07.739699 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:07.739663 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/588f099e-56f8-4f85-812b-174deb114247-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-mc2td\" (UID: \"588f099e-56f8-4f85-812b-174deb114247\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mc2td" Sep 10 16:11:07.739889 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:07.739745 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kqxv\" (UniqueName: \"kubernetes.io/projected/588f099e-56f8-4f85-812b-174deb114247-kube-api-access-8kqxv\") pod \"cert-manager-operator-controller-manager-55c444bf86-mc2td\" (UID: \"588f099e-56f8-4f85-812b-174deb114247\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mc2td" Sep 10 16:11:07.841280 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:07.841200 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kqxv\" (UniqueName: \"kubernetes.io/projected/588f099e-56f8-4f85-812b-174deb114247-kube-api-access-8kqxv\") pod \"cert-manager-operator-controller-manager-55c444bf86-mc2td\" (UID: \"588f099e-56f8-4f85-812b-174deb114247\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mc2td" Sep 10 16:11:07.841434 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:07.841283 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/588f099e-56f8-4f85-812b-174deb114247-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-mc2td\" (UID: \"588f099e-56f8-4f85-812b-174deb114247\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mc2td" Sep 10 16:11:07.841717 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:07.841689 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/588f099e-56f8-4f85-812b-174deb114247-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-mc2td\" (UID: \"588f099e-56f8-4f85-812b-174deb114247\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mc2td" Sep 10 16:11:07.850933 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:07.850899 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kqxv\" (UniqueName: \"kubernetes.io/projected/588f099e-56f8-4f85-812b-174deb114247-kube-api-access-8kqxv\") pod \"cert-manager-operator-controller-manager-55c444bf86-mc2td\" (UID: \"588f099e-56f8-4f85-812b-174deb114247\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mc2td" Sep 10 16:11:07.888966 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:07.888934 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mc2td" Sep 10 16:11:08.173045 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:08.173010 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mc2td"] Sep 10 16:11:08.176240 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:11:08.176212 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod588f099e_56f8_4f85_812b_174deb114247.slice/crio-66e89aff6e3914df2ecd1c68e66868530709950580ad946f2bf27d1ce3cc5c4c WatchSource:0}: Error finding container 66e89aff6e3914df2ecd1c68e66868530709950580ad946f2bf27d1ce3cc5c4c: Status 404 returned error can't find the container with id 66e89aff6e3914df2ecd1c68e66868530709950580ad946f2bf27d1ce3cc5c4c Sep 10 16:11:08.384508 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:08.384476 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mc2td" event={"ID":"588f099e-56f8-4f85-812b-174deb114247","Type":"ContainerStarted","Data":"66e89aff6e3914df2ecd1c68e66868530709950580ad946f2bf27d1ce3cc5c4c"} Sep 10 16:11:09.397290 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:09.397247 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:09.508837 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:09.508800 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:10.396227 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:10.396185 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mc2td" event={"ID":"588f099e-56f8-4f85-812b-174deb114247","Type":"ContainerStarted","Data":"bf04aea33c2c88d8fa77b016bbbccabee6fd25fb1e20a8456bb36cc16fd3f677"} Sep 10 16:11:10.423887 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:10.423826 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mc2td" podStartSLOduration=1.31431024 podStartE2EDuration="3.423804271s" totalImagesPullingTime="2.109494031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:11:10.422622887 +0000 UTC m=+569.945868084" watchObservedRunningTime="2026-09-10 16:11:10.423804271 +0000 UTC m=+569.947049469" Sep 10 16:11:11.292580 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:11.292543 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:11.577496 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:11.577417 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:11.637135 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:11.637100 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2"] Sep 10 16:11:11.640611 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:11.640595 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2" Sep 10 16:11:11.643674 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:11.643656 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hcb7s\"" Sep 10 16:11:11.643788 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:11.643689 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 16:11:11.644174 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:11.644148 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 16:11:11.656451 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:11.656425 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2"] Sep 10 16:11:11.778493 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:11.778449 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48629a9f-678a-4c79-a805-99938d152972-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2\" (UID: \"48629a9f-678a-4c79-a805-99938d152972\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2" Sep 10 16:11:11.778493 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:11.778499 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48629a9f-678a-4c79-a805-99938d152972-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2\" (UID: \"48629a9f-678a-4c79-a805-99938d152972\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2" Sep 10 16:11:11.778717 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:11.778568 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gfn9\" (UniqueName: \"kubernetes.io/projected/48629a9f-678a-4c79-a805-99938d152972-kube-api-access-8gfn9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2\" (UID: \"48629a9f-678a-4c79-a805-99938d152972\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2" Sep 10 16:11:11.879499 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:11.879468 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48629a9f-678a-4c79-a805-99938d152972-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2\" (UID: \"48629a9f-678a-4c79-a805-99938d152972\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2" Sep 10 16:11:11.879656 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:11.879509 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48629a9f-678a-4c79-a805-99938d152972-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2\" (UID: \"48629a9f-678a-4c79-a805-99938d152972\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2" Sep 10 16:11:11.879656 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:11.879551 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gfn9\" (UniqueName: \"kubernetes.io/projected/48629a9f-678a-4c79-a805-99938d152972-kube-api-access-8gfn9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2\" (UID: \"48629a9f-678a-4c79-a805-99938d152972\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2" Sep 10 16:11:11.879928 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:11.879906 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48629a9f-678a-4c79-a805-99938d152972-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2\" (UID: \"48629a9f-678a-4c79-a805-99938d152972\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2" Sep 10 16:11:11.879966 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:11.879916 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48629a9f-678a-4c79-a805-99938d152972-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2\" (UID: \"48629a9f-678a-4c79-a805-99938d152972\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2" Sep 10 16:11:11.891589 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:11.891568 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gfn9\" (UniqueName: \"kubernetes.io/projected/48629a9f-678a-4c79-a805-99938d152972-kube-api-access-8gfn9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2\" (UID: \"48629a9f-678a-4c79-a805-99938d152972\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2" Sep 10 16:11:11.949550 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:11.949515 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2" Sep 10 16:11:12.205893 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:12.205826 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kfxkc"] Sep 10 16:11:12.210216 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:12.210195 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-kfxkc" Sep 10 16:11:12.212603 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:12.212575 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 16:11:12.213113 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:12.213091 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 16:11:12.213209 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:12.213094 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-cw6h8\"" Sep 10 16:11:12.218189 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:12.218167 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kfxkc"] Sep 10 16:11:12.229857 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:12.229824 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2"] Sep 10 16:11:12.232895 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:11:12.232868 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48629a9f_678a_4c79_a805_99938d152972.slice/crio-054e8da6397cfd6c336182e12f6292b5111ad022e125d4312e0a66af925279bc WatchSource:0}: Error finding container 054e8da6397cfd6c336182e12f6292b5111ad022e125d4312e0a66af925279bc: Status 404 returned error can't find the container with id 054e8da6397cfd6c336182e12f6292b5111ad022e125d4312e0a66af925279bc Sep 10 16:11:12.383597 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:12.383569 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d9abd1d-cb84-4bde-ad7b-acb1c80ea113-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kfxkc\" (UID: \"0d9abd1d-cb84-4bde-ad7b-acb1c80ea113\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kfxkc" Sep 10 16:11:12.383782 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:12.383618 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcnjp\" (UniqueName: \"kubernetes.io/projected/0d9abd1d-cb84-4bde-ad7b-acb1c80ea113-kube-api-access-qcnjp\") pod \"cert-manager-webhook-6486ffd997-kfxkc\" (UID: \"0d9abd1d-cb84-4bde-ad7b-acb1c80ea113\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kfxkc" Sep 10 16:11:12.407397 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:12.407367 2541 generic.go:356] "Generic (PLEG): container finished" podID="48629a9f-678a-4c79-a805-99938d152972" containerID="2799023b88a3ef43c8107ddb0457bea54bf6d3290a8c36d0e94c08406cfde6b4" exitCode=0 Sep 10 16:11:12.407577 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:12.407442 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2" event={"ID":"48629a9f-678a-4c79-a805-99938d152972","Type":"ContainerDied","Data":"2799023b88a3ef43c8107ddb0457bea54bf6d3290a8c36d0e94c08406cfde6b4"} Sep 10 16:11:12.407577 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:12.407491 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2" event={"ID":"48629a9f-678a-4c79-a805-99938d152972","Type":"ContainerStarted","Data":"054e8da6397cfd6c336182e12f6292b5111ad022e125d4312e0a66af925279bc"} Sep 10 16:11:12.484448 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:12.484365 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d9abd1d-cb84-4bde-ad7b-acb1c80ea113-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kfxkc\" (UID: \"0d9abd1d-cb84-4bde-ad7b-acb1c80ea113\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kfxkc" Sep 10 16:11:12.484448 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:12.484411 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcnjp\" (UniqueName: \"kubernetes.io/projected/0d9abd1d-cb84-4bde-ad7b-acb1c80ea113-kube-api-access-qcnjp\") pod \"cert-manager-webhook-6486ffd997-kfxkc\" (UID: \"0d9abd1d-cb84-4bde-ad7b-acb1c80ea113\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kfxkc" Sep 10 16:11:12.493830 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:12.493810 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d9abd1d-cb84-4bde-ad7b-acb1c80ea113-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kfxkc\" (UID: \"0d9abd1d-cb84-4bde-ad7b-acb1c80ea113\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kfxkc" Sep 10 16:11:12.494065 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:12.494048 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcnjp\" (UniqueName: \"kubernetes.io/projected/0d9abd1d-cb84-4bde-ad7b-acb1c80ea113-kube-api-access-qcnjp\") pod \"cert-manager-webhook-6486ffd997-kfxkc\" (UID: \"0d9abd1d-cb84-4bde-ad7b-acb1c80ea113\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kfxkc" Sep 10 16:11:12.528805 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:12.528770 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-kfxkc" Sep 10 16:11:12.816397 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:12.816371 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kfxkc"] Sep 10 16:11:12.817620 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:11:12.817592 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d9abd1d_cb84_4bde_ad7b_acb1c80ea113.slice/crio-df1fd053b75026465dce74bc60ddf4995c591e38b67d71d3e27e59add5b7a341 WatchSource:0}: Error finding container df1fd053b75026465dce74bc60ddf4995c591e38b67d71d3e27e59add5b7a341: Status 404 returned error can't find the container with id df1fd053b75026465dce74bc60ddf4995c591e38b67d71d3e27e59add5b7a341 Sep 10 16:11:13.414278 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:13.414245 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-kfxkc" event={"ID":"0d9abd1d-cb84-4bde-ad7b-acb1c80ea113","Type":"ContainerStarted","Data":"df1fd053b75026465dce74bc60ddf4995c591e38b67d71d3e27e59add5b7a341"} Sep 10 16:11:13.605137 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:13.605105 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:14.039774 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:14.039737 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:15.436785 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:15.436745 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2" event={"ID":"48629a9f-678a-4c79-a805-99938d152972","Type":"ContainerStarted","Data":"0c4054696c842ce02f2dd442eed136672bc22d0bf9966903f55a65af5b3d41cc"} Sep 10 16:11:15.991146 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:15.991111 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:16.077022 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:16.076989 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:16.443973 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:16.443934 2541 generic.go:356] "Generic (PLEG): container finished" podID="48629a9f-678a-4c79-a805-99938d152972" containerID="0c4054696c842ce02f2dd442eed136672bc22d0bf9966903f55a65af5b3d41cc" exitCode=0 Sep 10 16:11:16.444405 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:16.444025 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2" event={"ID":"48629a9f-678a-4c79-a805-99938d152972","Type":"ContainerDied","Data":"0c4054696c842ce02f2dd442eed136672bc22d0bf9966903f55a65af5b3d41cc"} Sep 10 16:11:16.445602 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:16.445546 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-kfxkc" event={"ID":"0d9abd1d-cb84-4bde-ad7b-acb1c80ea113","Type":"ContainerStarted","Data":"dbc86fd2e4227b7b2b2a79c16321dcba95abfbea18dde9c76b25e200ff67b232"} Sep 10 16:11:16.445682 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:16.445622 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-kfxkc" Sep 10 16:11:16.490235 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:16.490189 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-kfxkc" podStartSLOduration=1.7157868939999998 podStartE2EDuration="4.490172145s" totalImagesPullingTime="2.774385251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:11:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:11:16.488387827 +0000 UTC m=+576.011633024" watchObservedRunningTime="2026-09-10 16:11:16.490172145 +0000 UTC m=+576.013417342" Sep 10 16:11:17.454698 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:17.454666 2541 generic.go:356] "Generic (PLEG): container finished" podID="48629a9f-678a-4c79-a805-99938d152972" containerID="00580650261f082c9158114c2bc7edb970e3dc3574f7393d9b10f7d9f1204713" exitCode=0 Sep 10 16:11:17.455080 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:17.454750 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2" event={"ID":"48629a9f-678a-4c79-a805-99938d152972","Type":"ContainerDied","Data":"00580650261f082c9158114c2bc7edb970e3dc3574f7393d9b10f7d9f1204713"} Sep 10 16:11:18.586764 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:18.586738 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2" Sep 10 16:11:18.634066 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:18.634036 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gfn9\" (UniqueName: \"kubernetes.io/projected/48629a9f-678a-4c79-a805-99938d152972-kube-api-access-8gfn9\") pod \"48629a9f-678a-4c79-a805-99938d152972\" (UID: \"48629a9f-678a-4c79-a805-99938d152972\") " Sep 10 16:11:18.634248 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:18.634082 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48629a9f-678a-4c79-a805-99938d152972-util\") pod \"48629a9f-678a-4c79-a805-99938d152972\" (UID: \"48629a9f-678a-4c79-a805-99938d152972\") " Sep 10 16:11:18.634248 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:18.634115 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48629a9f-678a-4c79-a805-99938d152972-bundle\") pod \"48629a9f-678a-4c79-a805-99938d152972\" (UID: \"48629a9f-678a-4c79-a805-99938d152972\") " Sep 10 16:11:18.634570 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:18.634543 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48629a9f-678a-4c79-a805-99938d152972-bundle" (OuterVolumeSpecName: "bundle") pod "48629a9f-678a-4c79-a805-99938d152972" (UID: "48629a9f-678a-4c79-a805-99938d152972"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:11:18.636388 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:18.636359 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48629a9f-678a-4c79-a805-99938d152972-kube-api-access-8gfn9" (OuterVolumeSpecName: "kube-api-access-8gfn9") pod "48629a9f-678a-4c79-a805-99938d152972" (UID: "48629a9f-678a-4c79-a805-99938d152972"). InnerVolumeSpecName "kube-api-access-8gfn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:11:18.638335 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:18.638316 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48629a9f-678a-4c79-a805-99938d152972-util" (OuterVolumeSpecName: "util") pod "48629a9f-678a-4c79-a805-99938d152972" (UID: "48629a9f-678a-4c79-a805-99938d152972"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:11:18.734935 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:18.734843 2541 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48629a9f-678a-4c79-a805-99938d152972-bundle\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:11:18.734935 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:18.734873 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gfn9\" (UniqueName: \"kubernetes.io/projected/48629a9f-678a-4c79-a805-99938d152972-kube-api-access-8gfn9\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:11:18.734935 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:18.734883 2541 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48629a9f-678a-4c79-a805-99938d152972-util\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:11:19.292568 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:19.292479 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vwzkb"] Sep 10 16:11:19.292804 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:19.292792 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48629a9f-678a-4c79-a805-99938d152972" containerName="pull" Sep 10 16:11:19.292853 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:19.292805 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="48629a9f-678a-4c79-a805-99938d152972" containerName="pull" Sep 10 16:11:19.292853 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:19.292820 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48629a9f-678a-4c79-a805-99938d152972" containerName="extract" Sep 10 16:11:19.292853 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:19.292825 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="48629a9f-678a-4c79-a805-99938d152972" containerName="extract" Sep 10 16:11:19.292853 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:19.292839 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48629a9f-678a-4c79-a805-99938d152972" containerName="util" Sep 10 16:11:19.292853 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:19.292844 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="48629a9f-678a-4c79-a805-99938d152972" containerName="util" Sep 10 16:11:19.293019 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:19.292892 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="48629a9f-678a-4c79-a805-99938d152972" containerName="extract" Sep 10 16:11:19.297178 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:19.297161 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vwzkb" Sep 10 16:11:19.299261 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:19.299234 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-pn4wd\"" Sep 10 16:11:19.304728 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:19.304703 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vwzkb"] Sep 10 16:11:19.338884 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:19.338859 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbfff322-877b-431b-848b-e4d20e079e3d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vwzkb\" (UID: \"bbfff322-877b-431b-848b-e4d20e079e3d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vwzkb" Sep 10 16:11:19.339022 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:19.338892 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqb8h\" (UniqueName: \"kubernetes.io/projected/bbfff322-877b-431b-848b-e4d20e079e3d-kube-api-access-vqb8h\") pod \"cert-manager-cainjector-559cdd8c87-vwzkb\" (UID: \"bbfff322-877b-431b-848b-e4d20e079e3d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vwzkb" Sep 10 16:11:19.439921 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:19.439886 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbfff322-877b-431b-848b-e4d20e079e3d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vwzkb\" (UID: \"bbfff322-877b-431b-848b-e4d20e079e3d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vwzkb" Sep 10 16:11:19.439921 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:19.439926 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqb8h\" (UniqueName: \"kubernetes.io/projected/bbfff322-877b-431b-848b-e4d20e079e3d-kube-api-access-vqb8h\") pod \"cert-manager-cainjector-559cdd8c87-vwzkb\" (UID: \"bbfff322-877b-431b-848b-e4d20e079e3d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vwzkb" Sep 10 16:11:19.452140 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:19.452099 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbfff322-877b-431b-848b-e4d20e079e3d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vwzkb\" (UID: \"bbfff322-877b-431b-848b-e4d20e079e3d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vwzkb" Sep 10 16:11:19.452289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:19.452266 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqb8h\" (UniqueName: \"kubernetes.io/projected/bbfff322-877b-431b-848b-e4d20e079e3d-kube-api-access-vqb8h\") pod \"cert-manager-cainjector-559cdd8c87-vwzkb\" (UID: \"bbfff322-877b-431b-848b-e4d20e079e3d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vwzkb" Sep 10 16:11:19.466758 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:19.466728 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2" event={"ID":"48629a9f-678a-4c79-a805-99938d152972","Type":"ContainerDied","Data":"054e8da6397cfd6c336182e12f6292b5111ad022e125d4312e0a66af925279bc"} Sep 10 16:11:19.466873 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:19.466768 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsbts2" Sep 10 16:11:19.466873 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:19.466780 2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="054e8da6397cfd6c336182e12f6292b5111ad022e125d4312e0a66af925279bc" Sep 10 16:11:19.606196 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:19.606108 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vwzkb" Sep 10 16:11:19.888668 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:19.888638 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vwzkb"] Sep 10 16:11:19.892405 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:11:19.892372 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbfff322_877b_431b_848b_e4d20e079e3d.slice/crio-afb10218847e856c8462d2f7d35d282b90a10f29fd39e793db3669ea349cf064 WatchSource:0}: Error finding container afb10218847e856c8462d2f7d35d282b90a10f29fd39e793db3669ea349cf064: Status 404 returned error can't find the container with id afb10218847e856c8462d2f7d35d282b90a10f29fd39e793db3669ea349cf064 Sep 10 16:11:20.472710 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:20.472677 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vwzkb" event={"ID":"bbfff322-877b-431b-848b-e4d20e079e3d","Type":"ContainerStarted","Data":"f9535ad116edfcafa0a51f4ff53202cf8b63b37ae1c1ece4280e32d12cda20b6"} Sep 10 16:11:20.472710 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:20.472716 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vwzkb" event={"ID":"bbfff322-877b-431b-848b-e4d20e079e3d","Type":"ContainerStarted","Data":"afb10218847e856c8462d2f7d35d282b90a10f29fd39e793db3669ea349cf064"} Sep 10 16:11:20.490555 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:20.490504 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vwzkb" podStartSLOduration=1.490487165 podStartE2EDuration="1.490487165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:11:20.48956428 +0000 UTC m=+580.012809482" watchObservedRunningTime="2026-09-10 16:11:20.490487165 +0000 UTC m=+580.013732363" Sep 10 16:11:20.704768 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:20.704731 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:20.849666 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:20.849576 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:21.845239 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:21.845202 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:22.457077 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:22.457050 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-kfxkc" Sep 10 16:11:24.034817 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:24.034782 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:24.227834 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:24.227801 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:30.313526 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:30.313493 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:30.443554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:30.443527 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:31.957958 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:31.957916 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:32.203713 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:32.203676 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9"] Sep 10 16:11:32.255760 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:32.255678 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9"] Sep 10 16:11:32.255907 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:32.255799 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9" Sep 10 16:11:32.258707 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:32.258688 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 16:11:32.258882 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:32.258689 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hcb7s\"" Sep 10 16:11:32.259015 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:32.258999 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 16:11:32.339148 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:32.339113 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5aa8a704-df3f-4a49-8af6-0da93a51e082-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9\" (UID: \"5aa8a704-df3f-4a49-8af6-0da93a51e082\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9" Sep 10 16:11:32.339148 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:32.339150 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5aa8a704-df3f-4a49-8af6-0da93a51e082-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9\" (UID: \"5aa8a704-df3f-4a49-8af6-0da93a51e082\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9" Sep 10 16:11:32.339387 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:32.339183 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdppj\" (UniqueName: \"kubernetes.io/projected/5aa8a704-df3f-4a49-8af6-0da93a51e082-kube-api-access-mdppj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9\" (UID: \"5aa8a704-df3f-4a49-8af6-0da93a51e082\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9" Sep 10 16:11:32.440480 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:32.440446 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5aa8a704-df3f-4a49-8af6-0da93a51e082-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9\" (UID: \"5aa8a704-df3f-4a49-8af6-0da93a51e082\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9" Sep 10 16:11:32.440683 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:32.440492 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5aa8a704-df3f-4a49-8af6-0da93a51e082-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9\" (UID: \"5aa8a704-df3f-4a49-8af6-0da93a51e082\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9" Sep 10 16:11:32.440683 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:32.440531 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdppj\" (UniqueName: \"kubernetes.io/projected/5aa8a704-df3f-4a49-8af6-0da93a51e082-kube-api-access-mdppj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9\" (UID: \"5aa8a704-df3f-4a49-8af6-0da93a51e082\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9" Sep 10 16:11:32.440927 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:32.440901 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5aa8a704-df3f-4a49-8af6-0da93a51e082-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9\" (UID: \"5aa8a704-df3f-4a49-8af6-0da93a51e082\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9" Sep 10 16:11:32.441006 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:32.440923 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5aa8a704-df3f-4a49-8af6-0da93a51e082-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9\" (UID: \"5aa8a704-df3f-4a49-8af6-0da93a51e082\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9" Sep 10 16:11:32.450659 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:32.450629 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdppj\" (UniqueName: \"kubernetes.io/projected/5aa8a704-df3f-4a49-8af6-0da93a51e082-kube-api-access-mdppj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9\" (UID: \"5aa8a704-df3f-4a49-8af6-0da93a51e082\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9" Sep 10 16:11:32.565575 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:32.565486 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9" Sep 10 16:11:32.860185 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:32.860112 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9"] Sep 10 16:11:32.863738 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:11:32.863709 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa8a704_df3f_4a49_8af6_0da93a51e082.slice/crio-a1c4eb65f541a1ca4224b4de351c0eeaeb32472a186942bcdc0527f0ca0a387b WatchSource:0}: Error finding container a1c4eb65f541a1ca4224b4de351c0eeaeb32472a186942bcdc0527f0ca0a387b: Status 404 returned error can't find the container with id a1c4eb65f541a1ca4224b4de351c0eeaeb32472a186942bcdc0527f0ca0a387b Sep 10 16:11:33.528721 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:33.528682 2541 generic.go:356] "Generic (PLEG): container finished" podID="5aa8a704-df3f-4a49-8af6-0da93a51e082" containerID="1d9072708eea329ba9bc1ea1c31dd69b474e047d536a113d29db8989c2f683c7" exitCode=0 Sep 10 16:11:33.529155 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:33.528770 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9" event={"ID":"5aa8a704-df3f-4a49-8af6-0da93a51e082","Type":"ContainerDied","Data":"1d9072708eea329ba9bc1ea1c31dd69b474e047d536a113d29db8989c2f683c7"} Sep 10 16:11:33.529155 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:33.528820 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9" event={"ID":"5aa8a704-df3f-4a49-8af6-0da93a51e082","Type":"ContainerStarted","Data":"a1c4eb65f541a1ca4224b4de351c0eeaeb32472a186942bcdc0527f0ca0a387b"} Sep 10 16:11:34.535036 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:34.534943 2541 generic.go:356] "Generic (PLEG): container finished" podID="5aa8a704-df3f-4a49-8af6-0da93a51e082" containerID="c9661a06200923f6f9d92343f376fade8671370073776c4b4366d82ec9d0d755" exitCode=0 Sep 10 16:11:34.535036 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:34.534996 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9" event={"ID":"5aa8a704-df3f-4a49-8af6-0da93a51e082","Type":"ContainerDied","Data":"c9661a06200923f6f9d92343f376fade8671370073776c4b4366d82ec9d0d755"} Sep 10 16:11:35.541318 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:35.541270 2541 generic.go:356] "Generic (PLEG): container finished" podID="5aa8a704-df3f-4a49-8af6-0da93a51e082" containerID="d403aa9d56626865a061fb3928746206e22d45261db38de8200154e01832fb4b" exitCode=0 Sep 10 16:11:35.541705 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:35.541334 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9" event={"ID":"5aa8a704-df3f-4a49-8af6-0da93a51e082","Type":"ContainerDied","Data":"d403aa9d56626865a061fb3928746206e22d45261db38de8200154e01832fb4b"} Sep 10 16:11:36.675145 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:36.675119 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9" Sep 10 16:11:36.775563 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:36.775517 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdppj\" (UniqueName: \"kubernetes.io/projected/5aa8a704-df3f-4a49-8af6-0da93a51e082-kube-api-access-mdppj\") pod \"5aa8a704-df3f-4a49-8af6-0da93a51e082\" (UID: \"5aa8a704-df3f-4a49-8af6-0da93a51e082\") " Sep 10 16:11:36.775563 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:36.775573 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5aa8a704-df3f-4a49-8af6-0da93a51e082-util\") pod \"5aa8a704-df3f-4a49-8af6-0da93a51e082\" (UID: \"5aa8a704-df3f-4a49-8af6-0da93a51e082\") " Sep 10 16:11:36.775785 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:36.775597 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5aa8a704-df3f-4a49-8af6-0da93a51e082-bundle\") pod \"5aa8a704-df3f-4a49-8af6-0da93a51e082\" (UID: \"5aa8a704-df3f-4a49-8af6-0da93a51e082\") " Sep 10 16:11:36.776355 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:36.776331 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa8a704-df3f-4a49-8af6-0da93a51e082-bundle" (OuterVolumeSpecName: "bundle") pod "5aa8a704-df3f-4a49-8af6-0da93a51e082" (UID: "5aa8a704-df3f-4a49-8af6-0da93a51e082"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:11:36.777864 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:36.777833 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa8a704-df3f-4a49-8af6-0da93a51e082-kube-api-access-mdppj" (OuterVolumeSpecName: "kube-api-access-mdppj") pod "5aa8a704-df3f-4a49-8af6-0da93a51e082" (UID: "5aa8a704-df3f-4a49-8af6-0da93a51e082"). InnerVolumeSpecName "kube-api-access-mdppj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:11:36.781123 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:36.781097 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa8a704-df3f-4a49-8af6-0da93a51e082-util" (OuterVolumeSpecName: "util") pod "5aa8a704-df3f-4a49-8af6-0da93a51e082" (UID: "5aa8a704-df3f-4a49-8af6-0da93a51e082"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:11:36.876730 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:36.876704 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdppj\" (UniqueName: \"kubernetes.io/projected/5aa8a704-df3f-4a49-8af6-0da93a51e082-kube-api-access-mdppj\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:11:36.876730 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:36.876733 2541 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5aa8a704-df3f-4a49-8af6-0da93a51e082-util\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:11:36.876730 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:36.876745 2541 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5aa8a704-df3f-4a49-8af6-0da93a51e082-bundle\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:11:37.557463 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:37.557425 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9" event={"ID":"5aa8a704-df3f-4a49-8af6-0da93a51e082","Type":"ContainerDied","Data":"a1c4eb65f541a1ca4224b4de351c0eeaeb32472a186942bcdc0527f0ca0a387b"} Sep 10 16:11:37.557463 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:37.557456 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afggjc9" Sep 10 16:11:37.557675 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:37.557495 2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1c4eb65f541a1ca4224b4de351c0eeaeb32472a186942bcdc0527f0ca0a387b" Sep 10 16:11:40.945173 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:40.945143 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/1.log" Sep 10 16:11:40.946256 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:40.946239 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/1.log" Sep 10 16:11:40.954125 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:40.954104 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4c6j_2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd/ovn-acl-logging/0.log" Sep 10 16:11:40.954967 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:40.954949 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4c6j_2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd/ovn-acl-logging/0.log" Sep 10 16:11:45.622434 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.622398 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss"] Sep 10 16:11:45.622808 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.622712 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aa8a704-df3f-4a49-8af6-0da93a51e082" containerName="extract" Sep 10 16:11:45.622808 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.622727 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aa8a704-df3f-4a49-8af6-0da93a51e082" containerName="extract" Sep 10 16:11:45.622808 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.622747 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aa8a704-df3f-4a49-8af6-0da93a51e082" containerName="util" Sep 10 16:11:45.622808 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.622752 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aa8a704-df3f-4a49-8af6-0da93a51e082" containerName="util" Sep 10 16:11:45.622808 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.622762 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aa8a704-df3f-4a49-8af6-0da93a51e082" containerName="pull" Sep 10 16:11:45.622808 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.622767 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aa8a704-df3f-4a49-8af6-0da93a51e082" containerName="pull" Sep 10 16:11:45.622985 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.622826 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="5aa8a704-df3f-4a49-8af6-0da93a51e082" containerName="extract" Sep 10 16:11:45.626971 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.626954 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss" Sep 10 16:11:45.629731 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.629707 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 16:11:45.629879 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.629863 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hcb7s\"" Sep 10 16:11:45.629919 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.629906 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 16:11:45.637545 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.637520 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss"] Sep 10 16:11:45.751880 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.751838 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zjvb\" (UniqueName: \"kubernetes.io/projected/3c636377-9bee-4658-a59b-4ff171fde71c-kube-api-access-4zjvb\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss\" (UID: \"3c636377-9bee-4658-a59b-4ff171fde71c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss" Sep 10 16:11:45.752074 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.751917 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c636377-9bee-4658-a59b-4ff171fde71c-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss\" (UID: \"3c636377-9bee-4658-a59b-4ff171fde71c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss" Sep 10 16:11:45.752074 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.751944 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c636377-9bee-4658-a59b-4ff171fde71c-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss\" (UID: \"3c636377-9bee-4658-a59b-4ff171fde71c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss" Sep 10 16:11:45.852473 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.852431 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c636377-9bee-4658-a59b-4ff171fde71c-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss\" (UID: \"3c636377-9bee-4658-a59b-4ff171fde71c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss" Sep 10 16:11:45.852668 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.852487 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c636377-9bee-4658-a59b-4ff171fde71c-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss\" (UID: \"3c636377-9bee-4658-a59b-4ff171fde71c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss" Sep 10 16:11:45.852668 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.852532 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zjvb\" (UniqueName: \"kubernetes.io/projected/3c636377-9bee-4658-a59b-4ff171fde71c-kube-api-access-4zjvb\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss\" (UID: \"3c636377-9bee-4658-a59b-4ff171fde71c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss" Sep 10 16:11:45.852867 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.852844 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c636377-9bee-4658-a59b-4ff171fde71c-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss\" (UID: \"3c636377-9bee-4658-a59b-4ff171fde71c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss" Sep 10 16:11:45.852930 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.852863 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c636377-9bee-4658-a59b-4ff171fde71c-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss\" (UID: \"3c636377-9bee-4658-a59b-4ff171fde71c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss" Sep 10 16:11:45.871149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.871111 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zjvb\" (UniqueName: \"kubernetes.io/projected/3c636377-9bee-4658-a59b-4ff171fde71c-kube-api-access-4zjvb\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss\" (UID: \"3c636377-9bee-4658-a59b-4ff171fde71c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss" Sep 10 16:11:45.936239 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:45.936149 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss" Sep 10 16:11:46.221126 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:46.221099 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss"] Sep 10 16:11:46.222922 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:11:46.222898 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c636377_9bee_4658_a59b_4ff171fde71c.slice/crio-94bc052cd33ffdb6f851a564c9f8e3d0f3481fc11fdfcf76385b88a92940ffe1 WatchSource:0}: Error finding container 94bc052cd33ffdb6f851a564c9f8e3d0f3481fc11fdfcf76385b88a92940ffe1: Status 404 returned error can't find the container with id 94bc052cd33ffdb6f851a564c9f8e3d0f3481fc11fdfcf76385b88a92940ffe1 Sep 10 16:11:46.600716 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:46.600615 2541 generic.go:356] "Generic (PLEG): container finished" podID="3c636377-9bee-4658-a59b-4ff171fde71c" containerID="b7751b13f64ad569372e973e402c8574ed59e32039b4bec0e965fd925ce6b5ab" exitCode=0 Sep 10 16:11:46.600716 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:46.600700 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss" event={"ID":"3c636377-9bee-4658-a59b-4ff171fde71c","Type":"ContainerDied","Data":"b7751b13f64ad569372e973e402c8574ed59e32039b4bec0e965fd925ce6b5ab"} Sep 10 16:11:46.600952 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:46.600748 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss" event={"ID":"3c636377-9bee-4658-a59b-4ff171fde71c","Type":"ContainerStarted","Data":"94bc052cd33ffdb6f851a564c9f8e3d0f3481fc11fdfcf76385b88a92940ffe1"} Sep 10 16:11:47.218934 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:47.218903 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:47.433080 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:47.432984 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:47.607422 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:47.607383 2541 generic.go:356] "Generic (PLEG): container finished" podID="3c636377-9bee-4658-a59b-4ff171fde71c" containerID="02cd572a0a089ce03b5c84164256deda08ca3f5b51eb5a0f0a6016f1f4c8c51a" exitCode=0 Sep 10 16:11:47.607609 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:47.607463 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss" event={"ID":"3c636377-9bee-4658-a59b-4ff171fde71c","Type":"ContainerDied","Data":"02cd572a0a089ce03b5c84164256deda08ca3f5b51eb5a0f0a6016f1f4c8c51a"} Sep 10 16:11:47.683265 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:47.683174 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:48.101164 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.101082 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:48.231765 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.231733 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh"] Sep 10 16:11:48.235093 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.235076 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" Sep 10 16:11:48.239401 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.239368 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 16:11:48.239401 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.239392 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9p2dd\"" Sep 10 16:11:48.239569 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.239412 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 16:11:48.239569 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.239488 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 16:11:48.239648 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.239617 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 16:11:48.299011 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.298984 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh"] Sep 10 16:11:48.314722 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.314698 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:48.370620 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.370555 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/235f85ce-8fc5-44fd-9110-0fa535c2f76e-manifests\") pod \"opendatahub-operator-controller-manager-67779d4d4c-6jlgh\" (UID: \"235f85ce-8fc5-44fd-9110-0fa535c2f76e\") " pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" Sep 10 16:11:48.370620 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.370599 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/235f85ce-8fc5-44fd-9110-0fa535c2f76e-tmp\") pod \"opendatahub-operator-controller-manager-67779d4d4c-6jlgh\" (UID: \"235f85ce-8fc5-44fd-9110-0fa535c2f76e\") " pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" Sep 10 16:11:48.370760 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.370628 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/235f85ce-8fc5-44fd-9110-0fa535c2f76e-webhook-cert\") pod \"opendatahub-operator-controller-manager-67779d4d4c-6jlgh\" (UID: \"235f85ce-8fc5-44fd-9110-0fa535c2f76e\") " pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" Sep 10 16:11:48.370760 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.370678 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/235f85ce-8fc5-44fd-9110-0fa535c2f76e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67779d4d4c-6jlgh\" (UID: \"235f85ce-8fc5-44fd-9110-0fa535c2f76e\") " pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" Sep 10 16:11:48.370760 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.370697 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6txgc\" (UniqueName: \"kubernetes.io/projected/235f85ce-8fc5-44fd-9110-0fa535c2f76e-kube-api-access-6txgc\") pod \"opendatahub-operator-controller-manager-67779d4d4c-6jlgh\" (UID: \"235f85ce-8fc5-44fd-9110-0fa535c2f76e\") " pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" Sep 10 16:11:48.471433 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.471405 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/235f85ce-8fc5-44fd-9110-0fa535c2f76e-tmp\") pod \"opendatahub-operator-controller-manager-67779d4d4c-6jlgh\" (UID: \"235f85ce-8fc5-44fd-9110-0fa535c2f76e\") " pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" Sep 10 16:11:48.471574 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.471444 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/235f85ce-8fc5-44fd-9110-0fa535c2f76e-webhook-cert\") pod \"opendatahub-operator-controller-manager-67779d4d4c-6jlgh\" (UID: \"235f85ce-8fc5-44fd-9110-0fa535c2f76e\") " pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" Sep 10 16:11:48.471574 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.471465 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/235f85ce-8fc5-44fd-9110-0fa535c2f76e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67779d4d4c-6jlgh\" (UID: \"235f85ce-8fc5-44fd-9110-0fa535c2f76e\") " pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" Sep 10 16:11:48.471574 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.471488 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6txgc\" (UniqueName: \"kubernetes.io/projected/235f85ce-8fc5-44fd-9110-0fa535c2f76e-kube-api-access-6txgc\") pod \"opendatahub-operator-controller-manager-67779d4d4c-6jlgh\" (UID: \"235f85ce-8fc5-44fd-9110-0fa535c2f76e\") " pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" Sep 10 16:11:48.471574 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.471545 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/235f85ce-8fc5-44fd-9110-0fa535c2f76e-manifests\") pod \"opendatahub-operator-controller-manager-67779d4d4c-6jlgh\" (UID: \"235f85ce-8fc5-44fd-9110-0fa535c2f76e\") " pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" Sep 10 16:11:48.471924 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.471894 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/235f85ce-8fc5-44fd-9110-0fa535c2f76e-manifests\") pod \"opendatahub-operator-controller-manager-67779d4d4c-6jlgh\" (UID: \"235f85ce-8fc5-44fd-9110-0fa535c2f76e\") " pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" Sep 10 16:11:48.473940 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.473917 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/235f85ce-8fc5-44fd-9110-0fa535c2f76e-tmp\") pod \"opendatahub-operator-controller-manager-67779d4d4c-6jlgh\" (UID: \"235f85ce-8fc5-44fd-9110-0fa535c2f76e\") " pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" Sep 10 16:11:48.474046 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.473965 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/235f85ce-8fc5-44fd-9110-0fa535c2f76e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67779d4d4c-6jlgh\" (UID: \"235f85ce-8fc5-44fd-9110-0fa535c2f76e\") " pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" Sep 10 16:11:48.474097 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.474080 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/235f85ce-8fc5-44fd-9110-0fa535c2f76e-webhook-cert\") pod \"opendatahub-operator-controller-manager-67779d4d4c-6jlgh\" (UID: \"235f85ce-8fc5-44fd-9110-0fa535c2f76e\") " pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" Sep 10 16:11:48.488628 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.488610 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6txgc\" (UniqueName: \"kubernetes.io/projected/235f85ce-8fc5-44fd-9110-0fa535c2f76e-kube-api-access-6txgc\") pod \"opendatahub-operator-controller-manager-67779d4d4c-6jlgh\" (UID: \"235f85ce-8fc5-44fd-9110-0fa535c2f76e\") " pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" Sep 10 16:11:48.544654 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.544626 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" Sep 10 16:11:48.615094 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.615065 2541 generic.go:356] "Generic (PLEG): container finished" podID="3c636377-9bee-4658-a59b-4ff171fde71c" containerID="3fb663810b78721234f1b03d94ecc7f883866f2988f2a29b7de8f6798258c5cc" exitCode=0 Sep 10 16:11:48.615202 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.615154 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss" event={"ID":"3c636377-9bee-4658-a59b-4ff171fde71c","Type":"ContainerDied","Data":"3fb663810b78721234f1b03d94ecc7f883866f2988f2a29b7de8f6798258c5cc"} Sep 10 16:11:48.778541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.778501 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:48.832559 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.832398 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh"] Sep 10 16:11:48.834891 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:11:48.834858 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235f85ce_8fc5_44fd_9110_0fa535c2f76e.slice/crio-4a491a024a26b6b36330e29ff3fbbf948fad1ac238e3b679f0526ac9157fe917 WatchSource:0}: Error finding container 4a491a024a26b6b36330e29ff3fbbf948fad1ac238e3b679f0526ac9157fe917: Status 404 returned error can't find the container with id 4a491a024a26b6b36330e29ff3fbbf948fad1ac238e3b679f0526ac9157fe917 Sep 10 16:11:48.909835 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:48.909802 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:49.019476 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:49.019441 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:49.118721 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:49.118649 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:49.248999 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:49.248965 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:49.623163 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:49.623085 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" event={"ID":"235f85ce-8fc5-44fd-9110-0fa535c2f76e","Type":"ContainerStarted","Data":"4a491a024a26b6b36330e29ff3fbbf948fad1ac238e3b679f0526ac9157fe917"} Sep 10 16:11:49.786636 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:49.786607 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss" Sep 10 16:11:49.883689 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:49.883578 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c636377-9bee-4658-a59b-4ff171fde71c-util\") pod \"3c636377-9bee-4658-a59b-4ff171fde71c\" (UID: \"3c636377-9bee-4658-a59b-4ff171fde71c\") " Sep 10 16:11:49.883689 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:49.883630 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c636377-9bee-4658-a59b-4ff171fde71c-bundle\") pod \"3c636377-9bee-4658-a59b-4ff171fde71c\" (UID: \"3c636377-9bee-4658-a59b-4ff171fde71c\") " Sep 10 16:11:49.883921 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:49.883690 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zjvb\" (UniqueName: \"kubernetes.io/projected/3c636377-9bee-4658-a59b-4ff171fde71c-kube-api-access-4zjvb\") pod \"3c636377-9bee-4658-a59b-4ff171fde71c\" (UID: \"3c636377-9bee-4658-a59b-4ff171fde71c\") " Sep 10 16:11:49.885044 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:49.885012 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c636377-9bee-4658-a59b-4ff171fde71c-bundle" (OuterVolumeSpecName: "bundle") pod "3c636377-9bee-4658-a59b-4ff171fde71c" (UID: "3c636377-9bee-4658-a59b-4ff171fde71c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:11:49.887221 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:49.887195 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c636377-9bee-4658-a59b-4ff171fde71c-kube-api-access-4zjvb" (OuterVolumeSpecName: "kube-api-access-4zjvb") pod "3c636377-9bee-4658-a59b-4ff171fde71c" (UID: "3c636377-9bee-4658-a59b-4ff171fde71c"). InnerVolumeSpecName "kube-api-access-4zjvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:11:49.892473 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:49.892429 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c636377-9bee-4658-a59b-4ff171fde71c-util" (OuterVolumeSpecName: "util") pod "3c636377-9bee-4658-a59b-4ff171fde71c" (UID: "3c636377-9bee-4658-a59b-4ff171fde71c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:11:49.984625 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:49.984589 2541 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c636377-9bee-4658-a59b-4ff171fde71c-bundle\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:11:49.984625 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:49.984619 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zjvb\" (UniqueName: \"kubernetes.io/projected/3c636377-9bee-4658-a59b-4ff171fde71c-kube-api-access-4zjvb\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:11:49.984625 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:49.984628 2541 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c636377-9bee-4658-a59b-4ff171fde71c-util\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:11:50.630271 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:50.630235 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss" event={"ID":"3c636377-9bee-4658-a59b-4ff171fde71c","Type":"ContainerDied","Data":"94bc052cd33ffdb6f851a564c9f8e3d0f3481fc11fdfcf76385b88a92940ffe1"} Sep 10 16:11:50.630271 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:50.630263 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hhcss" Sep 10 16:11:50.630830 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:50.630289 2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94bc052cd33ffdb6f851a564c9f8e3d0f3481fc11fdfcf76385b88a92940ffe1" Sep 10 16:11:52.641878 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:52.641839 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" event={"ID":"235f85ce-8fc5-44fd-9110-0fa535c2f76e","Type":"ContainerStarted","Data":"0cf6fb371bfe9ac1e7935b505f79268966be3ad1341ccb3f1e0ae078c090bbe4"} Sep 10 16:11:53.647912 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:53.647875 2541 generic.go:356] "Generic (PLEG): container finished" podID="235f85ce-8fc5-44fd-9110-0fa535c2f76e" containerID="0cf6fb371bfe9ac1e7935b505f79268966be3ad1341ccb3f1e0ae078c090bbe4" exitCode=0 Sep 10 16:11:53.648355 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:53.647961 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" event={"ID":"235f85ce-8fc5-44fd-9110-0fa535c2f76e","Type":"ContainerDied","Data":"0cf6fb371bfe9ac1e7935b505f79268966be3ad1341ccb3f1e0ae078c090bbe4"} Sep 10 16:11:54.654550 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:54.654517 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" event={"ID":"235f85ce-8fc5-44fd-9110-0fa535c2f76e","Type":"ContainerStarted","Data":"82cded301aad08828f9dfeca8a32720e5823bc5e6a549bd1236e260882350d41"} Sep 10 16:11:54.654928 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:54.654603 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" Sep 10 16:11:54.688772 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:11:54.688714 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" podStartSLOduration=2.229774869 podStartE2EDuration="6.688699666s" totalImagesPullingTime="3.382044696s" totalInitContainerRuntime="1.076880101s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:11:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:11:54.686215413 +0000 UTC m=+614.209460609" watchObservedRunningTime="2026-09-10 16:11:54.688699666 +0000 UTC m=+614.211944862" Sep 10 16:12:03.144854 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:03.144773 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:03.234454 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:03.234427 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:05.661914 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:05.661884 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67779d4d4c-6jlgh" Sep 10 16:12:08.357174 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.357141 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:08.491764 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.491733 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:08.736652 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.736618 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-74797cb87b-brnlx"] Sep 10 16:12:08.736979 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.736943 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c636377-9bee-4658-a59b-4ff171fde71c" containerName="pull" Sep 10 16:12:08.737029 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.736985 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c636377-9bee-4658-a59b-4ff171fde71c" containerName="pull" Sep 10 16:12:08.737029 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.737002 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c636377-9bee-4658-a59b-4ff171fde71c" containerName="util" Sep 10 16:12:08.737029 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.737007 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c636377-9bee-4658-a59b-4ff171fde71c" containerName="util" Sep 10 16:12:08.737029 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.737023 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c636377-9bee-4658-a59b-4ff171fde71c" containerName="extract" Sep 10 16:12:08.737029 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.737029 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c636377-9bee-4658-a59b-4ff171fde71c" containerName="extract" Sep 10 16:12:08.737177 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.737086 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c636377-9bee-4658-a59b-4ff171fde71c" containerName="extract" Sep 10 16:12:08.740101 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.740085 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-74797cb87b-brnlx" Sep 10 16:12:08.744401 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.744371 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 10 16:12:08.744576 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.744564 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 10 16:12:08.758929 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.758897 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:08.759648 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.759625 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-74797cb87b-brnlx"] Sep 10 16:12:08.855716 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.855682 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6238467c-baff-455b-a7fc-aa900e8f9fc2-tmp\") pod \"kube-auth-proxy-74797cb87b-brnlx\" (UID: \"6238467c-baff-455b-a7fc-aa900e8f9fc2\") " pod="openshift-ingress/kube-auth-proxy-74797cb87b-brnlx" Sep 10 16:12:08.855716 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.855719 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwhfz\" (UniqueName: \"kubernetes.io/projected/6238467c-baff-455b-a7fc-aa900e8f9fc2-kube-api-access-pwhfz\") pod \"kube-auth-proxy-74797cb87b-brnlx\" (UID: \"6238467c-baff-455b-a7fc-aa900e8f9fc2\") " pod="openshift-ingress/kube-auth-proxy-74797cb87b-brnlx" Sep 10 16:12:08.855921 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.855814 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6238467c-baff-455b-a7fc-aa900e8f9fc2-tls-certs\") pod \"kube-auth-proxy-74797cb87b-brnlx\" (UID: \"6238467c-baff-455b-a7fc-aa900e8f9fc2\") " pod="openshift-ingress/kube-auth-proxy-74797cb87b-brnlx" Sep 10 16:12:08.956577 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.956533 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6238467c-baff-455b-a7fc-aa900e8f9fc2-tmp\") pod \"kube-auth-proxy-74797cb87b-brnlx\" (UID: \"6238467c-baff-455b-a7fc-aa900e8f9fc2\") " pod="openshift-ingress/kube-auth-proxy-74797cb87b-brnlx" Sep 10 16:12:08.956766 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.956584 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwhfz\" (UniqueName: \"kubernetes.io/projected/6238467c-baff-455b-a7fc-aa900e8f9fc2-kube-api-access-pwhfz\") pod \"kube-auth-proxy-74797cb87b-brnlx\" (UID: \"6238467c-baff-455b-a7fc-aa900e8f9fc2\") " pod="openshift-ingress/kube-auth-proxy-74797cb87b-brnlx" Sep 10 16:12:08.956766 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.956662 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6238467c-baff-455b-a7fc-aa900e8f9fc2-tls-certs\") pod \"kube-auth-proxy-74797cb87b-brnlx\" (UID: \"6238467c-baff-455b-a7fc-aa900e8f9fc2\") " pod="openshift-ingress/kube-auth-proxy-74797cb87b-brnlx" Sep 10 16:12:08.956881 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:12:08.956788 2541 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 10 16:12:08.956881 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:12:08.956871 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6238467c-baff-455b-a7fc-aa900e8f9fc2-tls-certs podName:6238467c-baff-455b-a7fc-aa900e8f9fc2 nodeName:}" failed. No retries permitted until 2026-09-10 16:12:09.456850536 +0000 UTC m=+628.980095731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6238467c-baff-455b-a7fc-aa900e8f9fc2-tls-certs") pod "kube-auth-proxy-74797cb87b-brnlx" (UID: "6238467c-baff-455b-a7fc-aa900e8f9fc2") : secret "kube-auth-proxy-tls" not found Sep 10 16:12:08.958962 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.958944 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6238467c-baff-455b-a7fc-aa900e8f9fc2-tmp\") pod \"kube-auth-proxy-74797cb87b-brnlx\" (UID: \"6238467c-baff-455b-a7fc-aa900e8f9fc2\") " pod="openshift-ingress/kube-auth-proxy-74797cb87b-brnlx" Sep 10 16:12:08.975182 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:08.975156 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwhfz\" (UniqueName: \"kubernetes.io/projected/6238467c-baff-455b-a7fc-aa900e8f9fc2-kube-api-access-pwhfz\") pod \"kube-auth-proxy-74797cb87b-brnlx\" (UID: \"6238467c-baff-455b-a7fc-aa900e8f9fc2\") " pod="openshift-ingress/kube-auth-proxy-74797cb87b-brnlx" Sep 10 16:12:09.403760 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:09.403717 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:09.460889 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:09.460851 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6238467c-baff-455b-a7fc-aa900e8f9fc2-tls-certs\") pod \"kube-auth-proxy-74797cb87b-brnlx\" (UID: \"6238467c-baff-455b-a7fc-aa900e8f9fc2\") " pod="openshift-ingress/kube-auth-proxy-74797cb87b-brnlx" Sep 10 16:12:09.463433 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:09.463414 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6238467c-baff-455b-a7fc-aa900e8f9fc2-tls-certs\") pod \"kube-auth-proxy-74797cb87b-brnlx\" (UID: \"6238467c-baff-455b-a7fc-aa900e8f9fc2\") " pod="openshift-ingress/kube-auth-proxy-74797cb87b-brnlx" Sep 10 16:12:09.649487 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:09.649452 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-74797cb87b-brnlx" Sep 10 16:12:09.949003 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:09.948750 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-74797cb87b-brnlx"] Sep 10 16:12:09.955287 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:12:09.955263 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6238467c_baff_455b_a7fc_aa900e8f9fc2.slice/crio-0b2c35b29a8b62c51ddd015f4879c111b9d025930f9ef1a74e86fe3e94277bbc WatchSource:0}: Error finding container 0b2c35b29a8b62c51ddd015f4879c111b9d025930f9ef1a74e86fe3e94277bbc: Status 404 returned error can't find the container with id 0b2c35b29a8b62c51ddd015f4879c111b9d025930f9ef1a74e86fe3e94277bbc Sep 10 16:12:10.728491 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:10.728451 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-74797cb87b-brnlx" event={"ID":"6238467c-baff-455b-a7fc-aa900e8f9fc2","Type":"ContainerStarted","Data":"0b2c35b29a8b62c51ddd015f4879c111b9d025930f9ef1a74e86fe3e94277bbc"} Sep 10 16:12:11.197838 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.197801 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26"] Sep 10 16:12:11.201492 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.201472 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.204037 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.204004 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 16:12:11.204037 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.204020 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 16:12:11.204037 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.204004 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 16:12:11.204270 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.204055 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7rxbz\"" Sep 10 16:12:11.204530 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.204506 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 16:12:11.219320 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.219276 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26"] Sep 10 16:12:11.378409 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.378375 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zrb26\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.378601 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.378474 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgmqb\" (UniqueName: \"kubernetes.io/projected/ee343f95-7680-4ae4-be5b-6e53778d644d-kube-api-access-jgmqb\") pod \"istiod-openshift-gateway-7b7995d5cd-zrb26\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.378601 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.378565 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zrb26\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.378692 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.378618 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zrb26\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.378692 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.378675 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zrb26\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.378818 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.378797 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee343f95-7680-4ae4-be5b-6e53778d644d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zrb26\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.378944 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.378910 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee343f95-7680-4ae4-be5b-6e53778d644d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zrb26\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.479669 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.479589 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee343f95-7680-4ae4-be5b-6e53778d644d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zrb26\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.479669 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.479630 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee343f95-7680-4ae4-be5b-6e53778d644d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zrb26\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.479669 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.479647 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zrb26\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.479905 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.479768 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgmqb\" (UniqueName: \"kubernetes.io/projected/ee343f95-7680-4ae4-be5b-6e53778d644d-kube-api-access-jgmqb\") pod \"istiod-openshift-gateway-7b7995d5cd-zrb26\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.479905 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.479792 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zrb26\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.479905 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.479823 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zrb26\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.479905 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.479883 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zrb26\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.480605 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.480469 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zrb26\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.482084 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.482056 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee343f95-7680-4ae4-be5b-6e53778d644d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zrb26\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.482261 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.482246 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zrb26\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.482261 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.482252 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee343f95-7680-4ae4-be5b-6e53778d644d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zrb26\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.482365 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.482263 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zrb26\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.495140 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.495116 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgmqb\" (UniqueName: \"kubernetes.io/projected/ee343f95-7680-4ae4-be5b-6e53778d644d-kube-api-access-jgmqb\") pod \"istiod-openshift-gateway-7b7995d5cd-zrb26\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.495523 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.495508 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zrb26\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.498487 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.498458 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:11.514532 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.514511 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:11.597058 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.596986 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:11.805117 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:11.805027 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26"] Sep 10 16:12:12.427218 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:12:12.427185 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee343f95_7680_4ae4_be5b_6e53778d644d.slice/crio-72968a6b56ddb82ce2710b5fc73239659d563efb37aa241d5fd418a045e894b4 WatchSource:0}: Error finding container 72968a6b56ddb82ce2710b5fc73239659d563efb37aa241d5fd418a045e894b4: Status 404 returned error can't find the container with id 72968a6b56ddb82ce2710b5fc73239659d563efb37aa241d5fd418a045e894b4 Sep 10 16:12:12.558204 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:12.558172 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:12.741134 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:12.741041 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:12.741134 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:12.741069 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" event={"ID":"ee343f95-7680-4ae4-be5b-6e53778d644d","Type":"ContainerStarted","Data":"72968a6b56ddb82ce2710b5fc73239659d563efb37aa241d5fd418a045e894b4"} Sep 10 16:12:13.747441 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:13.747401 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-74797cb87b-brnlx" event={"ID":"6238467c-baff-455b-a7fc-aa900e8f9fc2","Type":"ContainerStarted","Data":"f1b47da965bf69bb27ff0564b367ebeffc04e1f4ddd2caa9e5b16048e09c30ed"} Sep 10 16:12:13.769629 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:13.769578 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-74797cb87b-brnlx" podStartSLOduration=2.411967761 podStartE2EDuration="5.76956085s" totalImagesPullingTime="3.357593089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:12:13.768094421 +0000 UTC m=+633.291339618" watchObservedRunningTime="2026-09-10 16:12:13.76956085 +0000 UTC m=+633.292806048" Sep 10 16:12:15.585963 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:15.585919 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:12:15.586232 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:15.585995 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:12:15.762421 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:15.762374 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" event={"ID":"ee343f95-7680-4ae4-be5b-6e53778d644d","Type":"ContainerStarted","Data":"cc85200e13de279a4e813621d39d1edcf279ac6c8216984bc08978a60c3bc83c"} Sep 10 16:12:15.762612 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:15.762514 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:15.789574 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:15.789410 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" podStartSLOduration=1.633033618 podStartE2EDuration="4.789388293s" totalImagesPullingTime="3.156354675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:12:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:12:15.788109316 +0000 UTC m=+635.311354528" watchObservedRunningTime="2026-09-10 16:12:15.789388293 +0000 UTC m=+635.312633491" Sep 10 16:12:16.429973 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:16.429925 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:16.769163 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:16.769068 2541 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-zrb26 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 16:12:16.769163 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:16.769129 2541 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" podUID="ee343f95-7680-4ae4-be5b-6e53778d644d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 16:12:17.358993 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:17.358946 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:17.590596 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:17.590563 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:17.769022 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:17.768988 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:19.532116 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:19.532075 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jd8qq"] Sep 10 16:12:19.535476 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:19.535461 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-jd8qq" Sep 10 16:12:19.537560 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:19.537537 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 10 16:12:19.537660 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:19.537540 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-r47n8\"" Sep 10 16:12:19.547055 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:19.547033 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jd8qq"] Sep 10 16:12:19.555672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:19.555647 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnxqk\" (UniqueName: \"kubernetes.io/projected/65e8d3c8-3d65-41c9-83f2-3e881d1639a9-kube-api-access-xnxqk\") pod \"odh-model-controller-7998c67dc4-jd8qq\" (UID: \"65e8d3c8-3d65-41c9-83f2-3e881d1639a9\") " pod="opendatahub/odh-model-controller-7998c67dc4-jd8qq" Sep 10 16:12:19.555801 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:19.555716 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65e8d3c8-3d65-41c9-83f2-3e881d1639a9-cert\") pod \"odh-model-controller-7998c67dc4-jd8qq\" (UID: \"65e8d3c8-3d65-41c9-83f2-3e881d1639a9\") " pod="opendatahub/odh-model-controller-7998c67dc4-jd8qq" Sep 10 16:12:19.573980 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:19.573936 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:19.656420 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:19.656384 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65e8d3c8-3d65-41c9-83f2-3e881d1639a9-cert\") pod \"odh-model-controller-7998c67dc4-jd8qq\" (UID: \"65e8d3c8-3d65-41c9-83f2-3e881d1639a9\") " pod="opendatahub/odh-model-controller-7998c67dc4-jd8qq" Sep 10 16:12:19.656620 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:19.656448 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnxqk\" (UniqueName: \"kubernetes.io/projected/65e8d3c8-3d65-41c9-83f2-3e881d1639a9-kube-api-access-xnxqk\") pod \"odh-model-controller-7998c67dc4-jd8qq\" (UID: \"65e8d3c8-3d65-41c9-83f2-3e881d1639a9\") " pod="opendatahub/odh-model-controller-7998c67dc4-jd8qq" Sep 10 16:12:19.656620 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:12:19.656537 2541 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 10 16:12:19.656620 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:12:19.656618 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65e8d3c8-3d65-41c9-83f2-3e881d1639a9-cert podName:65e8d3c8-3d65-41c9-83f2-3e881d1639a9 nodeName:}" failed. No retries permitted until 2026-09-10 16:12:20.156600416 +0000 UTC m=+639.679845591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/65e8d3c8-3d65-41c9-83f2-3e881d1639a9-cert") pod "odh-model-controller-7998c67dc4-jd8qq" (UID: "65e8d3c8-3d65-41c9-83f2-3e881d1639a9") : secret "odh-model-controller-webhook-cert" not found Sep 10 16:12:19.666283 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:19.666246 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:19.667865 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:19.667835 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnxqk\" (UniqueName: \"kubernetes.io/projected/65e8d3c8-3d65-41c9-83f2-3e881d1639a9-kube-api-access-xnxqk\") pod \"odh-model-controller-7998c67dc4-jd8qq\" (UID: \"65e8d3c8-3d65-41c9-83f2-3e881d1639a9\") " pod="opendatahub/odh-model-controller-7998c67dc4-jd8qq" Sep 10 16:12:19.767571 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:19.767533 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:19.768146 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:19.768123 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:19.864722 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:19.864649 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:20.115234 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:20.115149 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:20.161531 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:20.161500 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65e8d3c8-3d65-41c9-83f2-3e881d1639a9-cert\") pod \"odh-model-controller-7998c67dc4-jd8qq\" (UID: \"65e8d3c8-3d65-41c9-83f2-3e881d1639a9\") " pod="opendatahub/odh-model-controller-7998c67dc4-jd8qq" Sep 10 16:12:20.163922 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:20.163901 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65e8d3c8-3d65-41c9-83f2-3e881d1639a9-cert\") pod \"odh-model-controller-7998c67dc4-jd8qq\" (UID: \"65e8d3c8-3d65-41c9-83f2-3e881d1639a9\") " pod="opendatahub/odh-model-controller-7998c67dc4-jd8qq" Sep 10 16:12:20.437704 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:20.437669 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:20.445641 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:20.445620 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-jd8qq" Sep 10 16:12:20.733415 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:20.733363 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jd8qq"] Sep 10 16:12:20.736209 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:12:20.736187 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e8d3c8_3d65_41c9_83f2_3e881d1639a9.slice/crio-4b4e6a1884ceaa75ef07e4ca3b24b33115277bb473184f26e94b51e073554ffe WatchSource:0}: Error finding container 4b4e6a1884ceaa75ef07e4ca3b24b33115277bb473184f26e94b51e073554ffe: Status 404 returned error can't find the container with id 4b4e6a1884ceaa75ef07e4ca3b24b33115277bb473184f26e94b51e073554ffe Sep 10 16:12:20.787098 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:20.787056 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-jd8qq" event={"ID":"65e8d3c8-3d65-41c9-83f2-3e881d1639a9","Type":"ContainerStarted","Data":"4b4e6a1884ceaa75ef07e4ca3b24b33115277bb473184f26e94b51e073554ffe"} Sep 10 16:12:23.806542 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:23.806509 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-jd8qq" event={"ID":"65e8d3c8-3d65-41c9-83f2-3e881d1639a9","Type":"ContainerStarted","Data":"77acb427f3db8acf14c071fb990300a0b388e300e85ab0e13de058ee01cc07d2"} Sep 10 16:12:23.806934 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:23.806648 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-jd8qq" Sep 10 16:12:23.848052 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:23.847995 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-jd8qq" podStartSLOduration=1.91721946 podStartE2EDuration="4.847978075s" totalImagesPullingTime="2.930758615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:12:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:12:23.845164176 +0000 UTC m=+643.368409377" watchObservedRunningTime="2026-09-10 16:12:23.847978075 +0000 UTC m=+643.371223318" Sep 10 16:12:26.572374 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.572340 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2"] Sep 10 16:12:26.575820 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.575803 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:26.598703 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.598663 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2"] Sep 10 16:12:26.611643 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.611608 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1cf8d46-0b95-46ef-97d0-85545423d6e1-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-6bbs2\" (UID: \"e1cf8d46-0b95-46ef-97d0-85545423d6e1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:26.611643 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.611645 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e1cf8d46-0b95-46ef-97d0-85545423d6e1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-6bbs2\" (UID: \"e1cf8d46-0b95-46ef-97d0-85545423d6e1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:26.611879 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.611709 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e1cf8d46-0b95-46ef-97d0-85545423d6e1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-6bbs2\" (UID: \"e1cf8d46-0b95-46ef-97d0-85545423d6e1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:26.611879 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.611769 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfkjg\" (UniqueName: \"kubernetes.io/projected/e1cf8d46-0b95-46ef-97d0-85545423d6e1-kube-api-access-cfkjg\") pod \"istiod-openshift-gateway-5677ccb49b-6bbs2\" (UID: \"e1cf8d46-0b95-46ef-97d0-85545423d6e1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:26.611879 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.611791 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1cf8d46-0b95-46ef-97d0-85545423d6e1-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-6bbs2\" (UID: \"e1cf8d46-0b95-46ef-97d0-85545423d6e1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:26.611879 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.611850 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e1cf8d46-0b95-46ef-97d0-85545423d6e1-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-6bbs2\" (UID: \"e1cf8d46-0b95-46ef-97d0-85545423d6e1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:26.612112 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.611885 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1cf8d46-0b95-46ef-97d0-85545423d6e1-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-6bbs2\" (UID: \"e1cf8d46-0b95-46ef-97d0-85545423d6e1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:26.713250 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.713216 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfkjg\" (UniqueName: \"kubernetes.io/projected/e1cf8d46-0b95-46ef-97d0-85545423d6e1-kube-api-access-cfkjg\") pod \"istiod-openshift-gateway-5677ccb49b-6bbs2\" (UID: \"e1cf8d46-0b95-46ef-97d0-85545423d6e1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:26.713250 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.713253 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1cf8d46-0b95-46ef-97d0-85545423d6e1-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-6bbs2\" (UID: \"e1cf8d46-0b95-46ef-97d0-85545423d6e1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:26.713514 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.713278 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e1cf8d46-0b95-46ef-97d0-85545423d6e1-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-6bbs2\" (UID: \"e1cf8d46-0b95-46ef-97d0-85545423d6e1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:26.713514 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.713333 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1cf8d46-0b95-46ef-97d0-85545423d6e1-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-6bbs2\" (UID: \"e1cf8d46-0b95-46ef-97d0-85545423d6e1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:26.713514 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.713369 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1cf8d46-0b95-46ef-97d0-85545423d6e1-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-6bbs2\" (UID: \"e1cf8d46-0b95-46ef-97d0-85545423d6e1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:26.713514 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.713399 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e1cf8d46-0b95-46ef-97d0-85545423d6e1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-6bbs2\" (UID: \"e1cf8d46-0b95-46ef-97d0-85545423d6e1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:26.713694 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.713589 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e1cf8d46-0b95-46ef-97d0-85545423d6e1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-6bbs2\" (UID: \"e1cf8d46-0b95-46ef-97d0-85545423d6e1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:26.714573 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.714544 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e1cf8d46-0b95-46ef-97d0-85545423d6e1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-6bbs2\" (UID: \"e1cf8d46-0b95-46ef-97d0-85545423d6e1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:26.715824 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.715800 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1cf8d46-0b95-46ef-97d0-85545423d6e1-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-6bbs2\" (UID: \"e1cf8d46-0b95-46ef-97d0-85545423d6e1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:26.716006 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.715988 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1cf8d46-0b95-46ef-97d0-85545423d6e1-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-6bbs2\" (UID: \"e1cf8d46-0b95-46ef-97d0-85545423d6e1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:26.716085 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.716023 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e1cf8d46-0b95-46ef-97d0-85545423d6e1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-6bbs2\" (UID: \"e1cf8d46-0b95-46ef-97d0-85545423d6e1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:26.716145 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.716094 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e1cf8d46-0b95-46ef-97d0-85545423d6e1-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-6bbs2\" (UID: \"e1cf8d46-0b95-46ef-97d0-85545423d6e1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:26.722802 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.722776 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1cf8d46-0b95-46ef-97d0-85545423d6e1-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-6bbs2\" (UID: \"e1cf8d46-0b95-46ef-97d0-85545423d6e1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:26.723018 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.723001 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfkjg\" (UniqueName: \"kubernetes.io/projected/e1cf8d46-0b95-46ef-97d0-85545423d6e1-kube-api-access-cfkjg\") pod \"istiod-openshift-gateway-5677ccb49b-6bbs2\" (UID: \"e1cf8d46-0b95-46ef-97d0-85545423d6e1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:26.885404 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:26.885369 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:27.191692 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:27.191670 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2"] Sep 10 16:12:27.193710 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:12:27.193683 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1cf8d46_0b95_46ef_97d0_85545423d6e1.slice/crio-2a8d01bb6da777fef544efbf6e87bd69b4e25789f1488458d26e2e8156350072 WatchSource:0}: Error finding container 2a8d01bb6da777fef544efbf6e87bd69b4e25789f1488458d26e2e8156350072: Status 404 returned error can't find the container with id 2a8d01bb6da777fef544efbf6e87bd69b4e25789f1488458d26e2e8156350072 Sep 10 16:12:27.197115 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:27.197083 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:12:27.197183 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:27.197144 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:12:27.827454 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:27.827416 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" event={"ID":"e1cf8d46-0b95-46ef-97d0-85545423d6e1","Type":"ContainerStarted","Data":"26eacef6ae1dd2894dc41df9a47678f01b0a26de4e8cf7e986542fa852acca89"} Sep 10 16:12:27.827900 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:27.827504 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" event={"ID":"e1cf8d46-0b95-46ef-97d0-85545423d6e1","Type":"ContainerStarted","Data":"2a8d01bb6da777fef544efbf6e87bd69b4e25789f1488458d26e2e8156350072"} Sep 10 16:12:27.827900 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:27.827532 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:27.859089 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:27.859033 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" podStartSLOduration=1.859013762 podStartE2EDuration="1.859013762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:12:27.857086643 +0000 UTC m=+647.380331840" watchObservedRunningTime="2026-09-10 16:12:27.859013762 +0000 UTC m=+647.382258960" Sep 10 16:12:28.834157 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:28.834119 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6bbs2" Sep 10 16:12:28.979364 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:28.979335 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26"] Sep 10 16:12:28.979580 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:28.979537 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" podUID="ee343f95-7680-4ae4-be5b-6e53778d644d" containerName="discovery" containerID="cri-o://cc85200e13de279a4e813621d39d1edcf279ac6c8216984bc08978a60c3bc83c" gracePeriod=30 Sep 10 16:12:29.229463 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.229438 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:29.336834 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.336799 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-csr-ca-configmap\") pod \"ee343f95-7680-4ae4-be5b-6e53778d644d\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " Sep 10 16:12:29.336834 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.336839 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-csr-dns-cert\") pod \"ee343f95-7680-4ae4-be5b-6e53778d644d\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " Sep 10 16:12:29.337064 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.336954 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-kubeconfig\") pod \"ee343f95-7680-4ae4-be5b-6e53778d644d\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " Sep 10 16:12:29.337064 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.336987 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee343f95-7680-4ae4-be5b-6e53778d644d-local-certs\") pod \"ee343f95-7680-4ae4-be5b-6e53778d644d\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " Sep 10 16:12:29.337064 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.337012 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee343f95-7680-4ae4-be5b-6e53778d644d-cacerts\") pod \"ee343f95-7680-4ae4-be5b-6e53778d644d\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " Sep 10 16:12:29.337064 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.337049 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgmqb\" (UniqueName: \"kubernetes.io/projected/ee343f95-7680-4ae4-be5b-6e53778d644d-kube-api-access-jgmqb\") pod \"ee343f95-7680-4ae4-be5b-6e53778d644d\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " Sep 10 16:12:29.337064 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.337064 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-token\") pod \"ee343f95-7680-4ae4-be5b-6e53778d644d\" (UID: \"ee343f95-7680-4ae4-be5b-6e53778d644d\") " Sep 10 16:12:29.337358 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.337198 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "ee343f95-7680-4ae4-be5b-6e53778d644d" (UID: "ee343f95-7680-4ae4-be5b-6e53778d644d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:12:29.337470 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.337389 2541 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-csr-ca-configmap\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:12:29.339372 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.339348 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "ee343f95-7680-4ae4-be5b-6e53778d644d" (UID: "ee343f95-7680-4ae4-be5b-6e53778d644d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:12:29.339476 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.339384 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee343f95-7680-4ae4-be5b-6e53778d644d-cacerts" (OuterVolumeSpecName: "cacerts") pod "ee343f95-7680-4ae4-be5b-6e53778d644d" (UID: "ee343f95-7680-4ae4-be5b-6e53778d644d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:12:29.339824 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.339784 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee343f95-7680-4ae4-be5b-6e53778d644d-local-certs" (OuterVolumeSpecName: "local-certs") pod "ee343f95-7680-4ae4-be5b-6e53778d644d" (UID: "ee343f95-7680-4ae4-be5b-6e53778d644d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:12:29.339931 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.339859 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee343f95-7680-4ae4-be5b-6e53778d644d" (UID: "ee343f95-7680-4ae4-be5b-6e53778d644d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:12:29.339931 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.339882 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "ee343f95-7680-4ae4-be5b-6e53778d644d" (UID: "ee343f95-7680-4ae4-be5b-6e53778d644d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:12:29.339931 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.339885 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee343f95-7680-4ae4-be5b-6e53778d644d-kube-api-access-jgmqb" (OuterVolumeSpecName: "kube-api-access-jgmqb") pod "ee343f95-7680-4ae4-be5b-6e53778d644d" (UID: "ee343f95-7680-4ae4-be5b-6e53778d644d"). InnerVolumeSpecName "kube-api-access-jgmqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:12:29.438161 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.438126 2541 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-csr-dns-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:12:29.438161 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.438155 2541 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-kubeconfig\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:12:29.438161 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.438165 2541 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee343f95-7680-4ae4-be5b-6e53778d644d-local-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:12:29.438423 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.438173 2541 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee343f95-7680-4ae4-be5b-6e53778d644d-cacerts\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:12:29.438423 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.438181 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgmqb\" (UniqueName: \"kubernetes.io/projected/ee343f95-7680-4ae4-be5b-6e53778d644d-kube-api-access-jgmqb\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:12:29.438423 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.438188 2541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee343f95-7680-4ae4-be5b-6e53778d644d-istio-token\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:12:29.838379 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.838268 2541 generic.go:356] "Generic (PLEG): container finished" podID="ee343f95-7680-4ae4-be5b-6e53778d644d" containerID="cc85200e13de279a4e813621d39d1edcf279ac6c8216984bc08978a60c3bc83c" exitCode=0 Sep 10 16:12:29.838379 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.838329 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" Sep 10 16:12:29.838379 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.838360 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" event={"ID":"ee343f95-7680-4ae4-be5b-6e53778d644d","Type":"ContainerDied","Data":"cc85200e13de279a4e813621d39d1edcf279ac6c8216984bc08978a60c3bc83c"} Sep 10 16:12:29.838897 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.838413 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26" event={"ID":"ee343f95-7680-4ae4-be5b-6e53778d644d","Type":"ContainerDied","Data":"72968a6b56ddb82ce2710b5fc73239659d563efb37aa241d5fd418a045e894b4"} Sep 10 16:12:29.838897 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.838434 2541 scope.go:117] "RemoveContainer" containerID="cc85200e13de279a4e813621d39d1edcf279ac6c8216984bc08978a60c3bc83c" Sep 10 16:12:29.847757 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.847740 2541 scope.go:117] "RemoveContainer" containerID="cc85200e13de279a4e813621d39d1edcf279ac6c8216984bc08978a60c3bc83c" Sep 10 16:12:29.848003 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:12:29.847982 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc85200e13de279a4e813621d39d1edcf279ac6c8216984bc08978a60c3bc83c\": container with ID starting with cc85200e13de279a4e813621d39d1edcf279ac6c8216984bc08978a60c3bc83c not found: ID does not exist" containerID="cc85200e13de279a4e813621d39d1edcf279ac6c8216984bc08978a60c3bc83c" Sep 10 16:12:29.848067 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.848015 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc85200e13de279a4e813621d39d1edcf279ac6c8216984bc08978a60c3bc83c"} err="failed to get container status \"cc85200e13de279a4e813621d39d1edcf279ac6c8216984bc08978a60c3bc83c\": rpc error: code = NotFound desc = could not find container \"cc85200e13de279a4e813621d39d1edcf279ac6c8216984bc08978a60c3bc83c\": container with ID starting with cc85200e13de279a4e813621d39d1edcf279ac6c8216984bc08978a60c3bc83c not found: ID does not exist" Sep 10 16:12:29.887128 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.887095 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26"] Sep 10 16:12:29.888893 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.888873 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zrb26"] Sep 10 16:12:29.988390 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:29.988360 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:30.083125 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:30.083092 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-vncld"] Sep 10 16:12:30.083441 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:30.083429 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee343f95-7680-4ae4-be5b-6e53778d644d" containerName="discovery" Sep 10 16:12:30.083485 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:30.083443 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee343f95-7680-4ae4-be5b-6e53778d644d" containerName="discovery" Sep 10 16:12:30.083524 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:30.083515 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee343f95-7680-4ae4-be5b-6e53778d644d" containerName="discovery" Sep 10 16:12:30.087704 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:30.087681 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vncld" Sep 10 16:12:30.094613 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:30.094591 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-vnqhp\"" Sep 10 16:12:30.104743 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:30.104723 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 10 16:12:30.144834 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:30.144791 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr986\" (UniqueName: \"kubernetes.io/projected/fde2043f-9788-47c7-a46e-3ef85064a65c-kube-api-access-qr986\") pod \"kserve-controller-manager-d6f4b65b8-vncld\" (UID: \"fde2043f-9788-47c7-a46e-3ef85064a65c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vncld" Sep 10 16:12:30.145016 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:30.144879 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fde2043f-9788-47c7-a46e-3ef85064a65c-cert\") pod \"kserve-controller-manager-d6f4b65b8-vncld\" (UID: \"fde2043f-9788-47c7-a46e-3ef85064a65c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vncld" Sep 10 16:12:30.146078 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:30.146055 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-vncld"] Sep 10 16:12:30.215375 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:30.215339 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:30.245588 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:30.245557 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr986\" (UniqueName: \"kubernetes.io/projected/fde2043f-9788-47c7-a46e-3ef85064a65c-kube-api-access-qr986\") pod \"kserve-controller-manager-d6f4b65b8-vncld\" (UID: \"fde2043f-9788-47c7-a46e-3ef85064a65c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vncld" Sep 10 16:12:30.245717 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:30.245613 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fde2043f-9788-47c7-a46e-3ef85064a65c-cert\") pod \"kserve-controller-manager-d6f4b65b8-vncld\" (UID: \"fde2043f-9788-47c7-a46e-3ef85064a65c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vncld" Sep 10 16:12:30.248091 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:30.248073 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fde2043f-9788-47c7-a46e-3ef85064a65c-cert\") pod \"kserve-controller-manager-d6f4b65b8-vncld\" (UID: \"fde2043f-9788-47c7-a46e-3ef85064a65c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vncld" Sep 10 16:12:30.271764 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:30.271738 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr986\" (UniqueName: \"kubernetes.io/projected/fde2043f-9788-47c7-a46e-3ef85064a65c-kube-api-access-qr986\") pod \"kserve-controller-manager-d6f4b65b8-vncld\" (UID: \"fde2043f-9788-47c7-a46e-3ef85064a65c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vncld" Sep 10 16:12:30.397844 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:30.397815 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vncld" Sep 10 16:12:30.409848 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:30.409815 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:30.560805 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:30.560773 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:30.688689 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:30.688600 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-vncld"] Sep 10 16:12:30.700740 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:12:30.700717 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde2043f_9788_47c7_a46e_3ef85064a65c.slice/crio-1dc12ca58107d3102df5ca456d6c371a359b05bccdca57599a206c8c6b4979d6 WatchSource:0}: Error finding container 1dc12ca58107d3102df5ca456d6c371a359b05bccdca57599a206c8c6b4979d6: Status 404 returned error can't find the container with id 1dc12ca58107d3102df5ca456d6c371a359b05bccdca57599a206c8c6b4979d6 Sep 10 16:12:30.845254 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:30.845221 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vncld" event={"ID":"fde2043f-9788-47c7-a46e-3ef85064a65c","Type":"ContainerStarted","Data":"1dc12ca58107d3102df5ca456d6c371a359b05bccdca57599a206c8c6b4979d6"} Sep 10 16:12:30.984557 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:30.984476 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee343f95-7680-4ae4-be5b-6e53778d644d" path="/var/lib/kubelet/pods/ee343f95-7680-4ae4-be5b-6e53778d644d/volumes" Sep 10 16:12:33.864783 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:33.864746 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vncld" event={"ID":"fde2043f-9788-47c7-a46e-3ef85064a65c","Type":"ContainerStarted","Data":"41d88a34c7eb1fd0b916024f8ae89528270c6192118aaca5b9cc326a225754bc"} Sep 10 16:12:33.865235 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:33.864863 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vncld" Sep 10 16:12:33.899321 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:33.899238 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vncld" podStartSLOduration=1.489594284 podStartE2EDuration="3.899222185s" totalImagesPullingTime="2.409627901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:12:33.898349945 +0000 UTC m=+653.421595143" watchObservedRunningTime="2026-09-10 16:12:33.899222185 +0000 UTC m=+653.422467380" Sep 10 16:12:34.815047 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:12:34.814993 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-jd8qq" Sep 10 16:13:03.288338 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:03.288276 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:04.875353 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:04.875328 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vncld" Sep 10 16:13:05.043955 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:05.043925 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:05.119557 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:05.119520 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:20.395798 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:20.395759 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:20.494051 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:20.494010 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:21.060545 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:21.060506 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:21.133126 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:21.133082 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:26.038003 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.037941 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9"] Sep 10 16:13:26.041749 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.041726 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9" Sep 10 16:13:26.044572 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.044548 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hcb7s\"" Sep 10 16:13:26.044674 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.044609 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 16:13:26.045142 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.045127 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 16:13:26.056183 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.056161 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9"] Sep 10 16:13:26.132614 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.132582 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/465a84a6-6f3f-4c01-8311-2d56c1ea4611-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9\" (UID: \"465a84a6-6f3f-4c01-8311-2d56c1ea4611\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9" Sep 10 16:13:26.132771 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.132617 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mb5g\" (UniqueName: \"kubernetes.io/projected/465a84a6-6f3f-4c01-8311-2d56c1ea4611-kube-api-access-4mb5g\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9\" (UID: \"465a84a6-6f3f-4c01-8311-2d56c1ea4611\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9" Sep 10 16:13:26.132771 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.132747 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/465a84a6-6f3f-4c01-8311-2d56c1ea4611-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9\" (UID: \"465a84a6-6f3f-4c01-8311-2d56c1ea4611\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9" Sep 10 16:13:26.140073 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.140046 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw"] Sep 10 16:13:26.143373 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.143358 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw" Sep 10 16:13:26.155447 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.155426 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw"] Sep 10 16:13:26.233655 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.233620 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/465a84a6-6f3f-4c01-8311-2d56c1ea4611-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9\" (UID: \"465a84a6-6f3f-4c01-8311-2d56c1ea4611\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9" Sep 10 16:13:26.233823 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.233663 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77faf9bc-856f-40a9-8633-31b88cc8d8bf-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw\" (UID: \"77faf9bc-856f-40a9-8633-31b88cc8d8bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw" Sep 10 16:13:26.233823 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.233684 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77faf9bc-856f-40a9-8633-31b88cc8d8bf-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw\" (UID: \"77faf9bc-856f-40a9-8633-31b88cc8d8bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw" Sep 10 16:13:26.233823 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.233711 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/465a84a6-6f3f-4c01-8311-2d56c1ea4611-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9\" (UID: \"465a84a6-6f3f-4c01-8311-2d56c1ea4611\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9" Sep 10 16:13:26.233823 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.233733 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mb5g\" (UniqueName: \"kubernetes.io/projected/465a84a6-6f3f-4c01-8311-2d56c1ea4611-kube-api-access-4mb5g\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9\" (UID: \"465a84a6-6f3f-4c01-8311-2d56c1ea4611\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9" Sep 10 16:13:26.233823 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.233784 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lzr9\" (UniqueName: \"kubernetes.io/projected/77faf9bc-856f-40a9-8633-31b88cc8d8bf-kube-api-access-5lzr9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw\" (UID: \"77faf9bc-856f-40a9-8633-31b88cc8d8bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw" Sep 10 16:13:26.234094 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.234073 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/465a84a6-6f3f-4c01-8311-2d56c1ea4611-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9\" (UID: \"465a84a6-6f3f-4c01-8311-2d56c1ea4611\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9" Sep 10 16:13:26.234159 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.234073 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/465a84a6-6f3f-4c01-8311-2d56c1ea4611-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9\" (UID: \"465a84a6-6f3f-4c01-8311-2d56c1ea4611\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9" Sep 10 16:13:26.236801 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.236778 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9"] Sep 10 16:13:26.240267 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.240253 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9" Sep 10 16:13:26.246888 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.246859 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mb5g\" (UniqueName: \"kubernetes.io/projected/465a84a6-6f3f-4c01-8311-2d56c1ea4611-kube-api-access-4mb5g\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9\" (UID: \"465a84a6-6f3f-4c01-8311-2d56c1ea4611\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9" Sep 10 16:13:26.255971 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.255946 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9"] Sep 10 16:13:26.334735 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.334647 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77faf9bc-856f-40a9-8633-31b88cc8d8bf-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw\" (UID: \"77faf9bc-856f-40a9-8633-31b88cc8d8bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw" Sep 10 16:13:26.334735 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.334691 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77faf9bc-856f-40a9-8633-31b88cc8d8bf-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw\" (UID: \"77faf9bc-856f-40a9-8633-31b88cc8d8bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw" Sep 10 16:13:26.334735 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.334728 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/41fd568a-c802-4edf-9ee8-66b427a77df2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9\" (UID: \"41fd568a-c802-4edf-9ee8-66b427a77df2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9" Sep 10 16:13:26.334963 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.334773 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lzr9\" (UniqueName: \"kubernetes.io/projected/77faf9bc-856f-40a9-8633-31b88cc8d8bf-kube-api-access-5lzr9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw\" (UID: \"77faf9bc-856f-40a9-8633-31b88cc8d8bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw" Sep 10 16:13:26.334963 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.334809 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlqrc\" (UniqueName: \"kubernetes.io/projected/41fd568a-c802-4edf-9ee8-66b427a77df2-kube-api-access-qlqrc\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9\" (UID: \"41fd568a-c802-4edf-9ee8-66b427a77df2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9" Sep 10 16:13:26.334963 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.334859 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/41fd568a-c802-4edf-9ee8-66b427a77df2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9\" (UID: \"41fd568a-c802-4edf-9ee8-66b427a77df2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9" Sep 10 16:13:26.335104 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.335075 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77faf9bc-856f-40a9-8633-31b88cc8d8bf-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw\" (UID: \"77faf9bc-856f-40a9-8633-31b88cc8d8bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw" Sep 10 16:13:26.335146 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.335076 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77faf9bc-856f-40a9-8633-31b88cc8d8bf-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw\" (UID: \"77faf9bc-856f-40a9-8633-31b88cc8d8bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw" Sep 10 16:13:26.338342 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.338317 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc"] Sep 10 16:13:26.342257 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.342239 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc" Sep 10 16:13:26.349030 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.349000 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lzr9\" (UniqueName: \"kubernetes.io/projected/77faf9bc-856f-40a9-8633-31b88cc8d8bf-kube-api-access-5lzr9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw\" (UID: \"77faf9bc-856f-40a9-8633-31b88cc8d8bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw" Sep 10 16:13:26.350527 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.350510 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9" Sep 10 16:13:26.360707 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.360686 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc"] Sep 10 16:13:26.436008 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.435976 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twtl6\" (UniqueName: \"kubernetes.io/projected/6b5aaf3b-819c-4e40-8068-0d93b420ed05-kube-api-access-twtl6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc\" (UID: \"6b5aaf3b-819c-4e40-8068-0d93b420ed05\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc" Sep 10 16:13:26.436181 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.436027 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlqrc\" (UniqueName: \"kubernetes.io/projected/41fd568a-c802-4edf-9ee8-66b427a77df2-kube-api-access-qlqrc\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9\" (UID: \"41fd568a-c802-4edf-9ee8-66b427a77df2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9" Sep 10 16:13:26.436181 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.436075 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/41fd568a-c802-4edf-9ee8-66b427a77df2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9\" (UID: \"41fd568a-c802-4edf-9ee8-66b427a77df2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9" Sep 10 16:13:26.436181 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.436137 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b5aaf3b-819c-4e40-8068-0d93b420ed05-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc\" (UID: \"6b5aaf3b-819c-4e40-8068-0d93b420ed05\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc" Sep 10 16:13:26.436373 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.436185 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/41fd568a-c802-4edf-9ee8-66b427a77df2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9\" (UID: \"41fd568a-c802-4edf-9ee8-66b427a77df2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9" Sep 10 16:13:26.436373 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.436208 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b5aaf3b-819c-4e40-8068-0d93b420ed05-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc\" (UID: \"6b5aaf3b-819c-4e40-8068-0d93b420ed05\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc" Sep 10 16:13:26.436545 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.436517 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/41fd568a-c802-4edf-9ee8-66b427a77df2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9\" (UID: \"41fd568a-c802-4edf-9ee8-66b427a77df2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9" Sep 10 16:13:26.436608 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.436576 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/41fd568a-c802-4edf-9ee8-66b427a77df2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9\" (UID: \"41fd568a-c802-4edf-9ee8-66b427a77df2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9" Sep 10 16:13:26.447600 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.447563 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlqrc\" (UniqueName: \"kubernetes.io/projected/41fd568a-c802-4edf-9ee8-66b427a77df2-kube-api-access-qlqrc\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9\" (UID: \"41fd568a-c802-4edf-9ee8-66b427a77df2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9" Sep 10 16:13:26.453603 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.453576 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw" Sep 10 16:13:26.537116 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.537079 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b5aaf3b-819c-4e40-8068-0d93b420ed05-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc\" (UID: \"6b5aaf3b-819c-4e40-8068-0d93b420ed05\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc" Sep 10 16:13:26.537270 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.537138 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b5aaf3b-819c-4e40-8068-0d93b420ed05-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc\" (UID: \"6b5aaf3b-819c-4e40-8068-0d93b420ed05\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc" Sep 10 16:13:26.537270 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.537175 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twtl6\" (UniqueName: \"kubernetes.io/projected/6b5aaf3b-819c-4e40-8068-0d93b420ed05-kube-api-access-twtl6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc\" (UID: \"6b5aaf3b-819c-4e40-8068-0d93b420ed05\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc" Sep 10 16:13:26.537556 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.537534 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b5aaf3b-819c-4e40-8068-0d93b420ed05-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc\" (UID: \"6b5aaf3b-819c-4e40-8068-0d93b420ed05\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc" Sep 10 16:13:26.537597 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.537537 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b5aaf3b-819c-4e40-8068-0d93b420ed05-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc\" (UID: \"6b5aaf3b-819c-4e40-8068-0d93b420ed05\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc" Sep 10 16:13:26.548815 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.548789 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twtl6\" (UniqueName: \"kubernetes.io/projected/6b5aaf3b-819c-4e40-8068-0d93b420ed05-kube-api-access-twtl6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc\" (UID: \"6b5aaf3b-819c-4e40-8068-0d93b420ed05\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc" Sep 10 16:13:26.557629 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.557609 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9" Sep 10 16:13:26.638683 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.638652 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9"] Sep 10 16:13:26.639643 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:13:26.639623 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465a84a6_6f3f_4c01_8311_2d56c1ea4611.slice/crio-b2718ef86aa34a209e03bd931b295ae33bd862bd75775e73c96ea50de8430596 WatchSource:0}: Error finding container b2718ef86aa34a209e03bd931b295ae33bd862bd75775e73c96ea50de8430596: Status 404 returned error can't find the container with id b2718ef86aa34a209e03bd931b295ae33bd862bd75775e73c96ea50de8430596 Sep 10 16:13:26.660514 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.660495 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc" Sep 10 16:13:26.729854 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.729825 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw"] Sep 10 16:13:26.732575 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:13:26.732548 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77faf9bc_856f_40a9_8633_31b88cc8d8bf.slice/crio-03103ee686297dbcb71227efe976d31504b116b2b64bd9738b14f7ba722a8aea WatchSource:0}: Error finding container 03103ee686297dbcb71227efe976d31504b116b2b64bd9738b14f7ba722a8aea: Status 404 returned error can't find the container with id 03103ee686297dbcb71227efe976d31504b116b2b64bd9738b14f7ba722a8aea Sep 10 16:13:26.840372 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.840348 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9"] Sep 10 16:13:26.841481 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:13:26.841460 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd568a_c802_4edf_9ee8_66b427a77df2.slice/crio-e595ba4ed05e3e84a74403bd40fa91541e3bb0c82118346b72c24b6c4775289b WatchSource:0}: Error finding container e595ba4ed05e3e84a74403bd40fa91541e3bb0c82118346b72c24b6c4775289b: Status 404 returned error can't find the container with id e595ba4ed05e3e84a74403bd40fa91541e3bb0c82118346b72c24b6c4775289b Sep 10 16:13:26.941723 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:26.941696 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc"] Sep 10 16:13:26.982234 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:13:26.982198 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5aaf3b_819c_4e40_8068_0d93b420ed05.slice/crio-c575c576d2adc0807b83975bbdf03f3b1a54960a90f5ba5a8bcc21930994118e WatchSource:0}: Error finding container c575c576d2adc0807b83975bbdf03f3b1a54960a90f5ba5a8bcc21930994118e: Status 404 returned error can't find the container with id c575c576d2adc0807b83975bbdf03f3b1a54960a90f5ba5a8bcc21930994118e Sep 10 16:13:27.133099 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:27.133064 2541 generic.go:356] "Generic (PLEG): container finished" podID="77faf9bc-856f-40a9-8633-31b88cc8d8bf" containerID="8a244325e7a3ec8530284e18c564862c0617618761a314401d366834a10206d7" exitCode=0 Sep 10 16:13:27.133550 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:27.133144 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw" event={"ID":"77faf9bc-856f-40a9-8633-31b88cc8d8bf","Type":"ContainerDied","Data":"8a244325e7a3ec8530284e18c564862c0617618761a314401d366834a10206d7"} Sep 10 16:13:27.133550 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:27.133192 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw" event={"ID":"77faf9bc-856f-40a9-8633-31b88cc8d8bf","Type":"ContainerStarted","Data":"03103ee686297dbcb71227efe976d31504b116b2b64bd9738b14f7ba722a8aea"} Sep 10 16:13:27.135023 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:27.134999 2541 generic.go:356] "Generic (PLEG): container finished" podID="465a84a6-6f3f-4c01-8311-2d56c1ea4611" containerID="92bfef38a4e8d8a58eaadde28a8fee9eb83302bb4c58b4fc19fe47195eb5a8d0" exitCode=0 Sep 10 16:13:27.135171 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:27.135087 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9" event={"ID":"465a84a6-6f3f-4c01-8311-2d56c1ea4611","Type":"ContainerDied","Data":"92bfef38a4e8d8a58eaadde28a8fee9eb83302bb4c58b4fc19fe47195eb5a8d0"} Sep 10 16:13:27.135171 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:27.135139 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9" event={"ID":"465a84a6-6f3f-4c01-8311-2d56c1ea4611","Type":"ContainerStarted","Data":"b2718ef86aa34a209e03bd931b295ae33bd862bd75775e73c96ea50de8430596"} Sep 10 16:13:27.136728 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:27.136706 2541 generic.go:356] "Generic (PLEG): container finished" podID="6b5aaf3b-819c-4e40-8068-0d93b420ed05" containerID="551767afe42fd7fe0712241221a88c8db1178d323abad8942906a3f285caaa47" exitCode=0 Sep 10 16:13:27.136826 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:27.136758 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc" event={"ID":"6b5aaf3b-819c-4e40-8068-0d93b420ed05","Type":"ContainerDied","Data":"551767afe42fd7fe0712241221a88c8db1178d323abad8942906a3f285caaa47"} Sep 10 16:13:27.136826 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:27.136800 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc" event={"ID":"6b5aaf3b-819c-4e40-8068-0d93b420ed05","Type":"ContainerStarted","Data":"c575c576d2adc0807b83975bbdf03f3b1a54960a90f5ba5a8bcc21930994118e"} Sep 10 16:13:27.138686 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:27.138666 2541 generic.go:356] "Generic (PLEG): container finished" podID="41fd568a-c802-4edf-9ee8-66b427a77df2" containerID="eaaf674c760ea9a1e4bc24c7d75751ee8bdc95c2a0c8c1fd95373c908d7f3a89" exitCode=0 Sep 10 16:13:27.138776 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:27.138734 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9" event={"ID":"41fd568a-c802-4edf-9ee8-66b427a77df2","Type":"ContainerDied","Data":"eaaf674c760ea9a1e4bc24c7d75751ee8bdc95c2a0c8c1fd95373c908d7f3a89"} Sep 10 16:13:27.138776 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:27.138771 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9" event={"ID":"41fd568a-c802-4edf-9ee8-66b427a77df2","Type":"ContainerStarted","Data":"e595ba4ed05e3e84a74403bd40fa91541e3bb0c82118346b72c24b6c4775289b"} Sep 10 16:13:28.146179 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:28.146146 2541 generic.go:356] "Generic (PLEG): container finished" podID="6b5aaf3b-819c-4e40-8068-0d93b420ed05" containerID="3f5079a2b803b979094f09a49b4daa1345ff722a56d0dd16c329ed6155d7547e" exitCode=0 Sep 10 16:13:28.146621 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:28.146248 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc" event={"ID":"6b5aaf3b-819c-4e40-8068-0d93b420ed05","Type":"ContainerDied","Data":"3f5079a2b803b979094f09a49b4daa1345ff722a56d0dd16c329ed6155d7547e"} Sep 10 16:13:28.148278 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:28.148259 2541 generic.go:356] "Generic (PLEG): container finished" podID="41fd568a-c802-4edf-9ee8-66b427a77df2" containerID="91e6d02876ff3dabece7c468a016132eeac0d9efb3092f33811f8abf1db64236" exitCode=0 Sep 10 16:13:28.148422 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:28.148363 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9" event={"ID":"41fd568a-c802-4edf-9ee8-66b427a77df2","Type":"ContainerDied","Data":"91e6d02876ff3dabece7c468a016132eeac0d9efb3092f33811f8abf1db64236"} Sep 10 16:13:29.160636 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:29.160602 2541 generic.go:356] "Generic (PLEG): container finished" podID="77faf9bc-856f-40a9-8633-31b88cc8d8bf" containerID="c8efd831b8e2ee9b605c3413b74a0e1ffa64394e120d741e793aff7eb6247227" exitCode=0 Sep 10 16:13:29.161053 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:29.160678 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw" event={"ID":"77faf9bc-856f-40a9-8633-31b88cc8d8bf","Type":"ContainerDied","Data":"c8efd831b8e2ee9b605c3413b74a0e1ffa64394e120d741e793aff7eb6247227"} Sep 10 16:13:29.162578 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:29.162557 2541 generic.go:356] "Generic (PLEG): container finished" podID="465a84a6-6f3f-4c01-8311-2d56c1ea4611" containerID="ef92a962d944aca7510406a6a8d2cf60b2d8df3a45710833f27b022dc3142d0b" exitCode=0 Sep 10 16:13:29.162669 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:29.162628 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9" event={"ID":"465a84a6-6f3f-4c01-8311-2d56c1ea4611","Type":"ContainerDied","Data":"ef92a962d944aca7510406a6a8d2cf60b2d8df3a45710833f27b022dc3142d0b"} Sep 10 16:13:29.165047 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:29.165024 2541 generic.go:356] "Generic (PLEG): container finished" podID="6b5aaf3b-819c-4e40-8068-0d93b420ed05" containerID="33555ca69a79ad91cfc9ffc51e731dd74ee0c5bdf1db47c73a46a49c853aaa3d" exitCode=0 Sep 10 16:13:29.165132 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:29.165093 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc" event={"ID":"6b5aaf3b-819c-4e40-8068-0d93b420ed05","Type":"ContainerDied","Data":"33555ca69a79ad91cfc9ffc51e731dd74ee0c5bdf1db47c73a46a49c853aaa3d"} Sep 10 16:13:29.167682 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:29.167659 2541 generic.go:356] "Generic (PLEG): container finished" podID="41fd568a-c802-4edf-9ee8-66b427a77df2" containerID="cadc16968b6fccf2a7feb0093f36059ea0b30954dd13f084b3b492f92abfe78d" exitCode=0 Sep 10 16:13:29.167785 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:29.167746 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9" event={"ID":"41fd568a-c802-4edf-9ee8-66b427a77df2","Type":"ContainerDied","Data":"cadc16968b6fccf2a7feb0093f36059ea0b30954dd13f084b3b492f92abfe78d"} Sep 10 16:13:30.175482 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.175447 2541 generic.go:356] "Generic (PLEG): container finished" podID="77faf9bc-856f-40a9-8633-31b88cc8d8bf" containerID="44024c19d0cecd9a5287987552c6ea8e45b3753efe3155f451688503613be5d4" exitCode=0 Sep 10 16:13:30.175873 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.175527 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw" event={"ID":"77faf9bc-856f-40a9-8633-31b88cc8d8bf","Type":"ContainerDied","Data":"44024c19d0cecd9a5287987552c6ea8e45b3753efe3155f451688503613be5d4"} Sep 10 16:13:30.177565 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.177540 2541 generic.go:356] "Generic (PLEG): container finished" podID="465a84a6-6f3f-4c01-8311-2d56c1ea4611" containerID="7ac739e8eddc8d74450ade71d76c47e190e016b4f890d58cb14b5e99947a2f17" exitCode=0 Sep 10 16:13:30.177661 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.177615 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9" event={"ID":"465a84a6-6f3f-4c01-8311-2d56c1ea4611","Type":"ContainerDied","Data":"7ac739e8eddc8d74450ade71d76c47e190e016b4f890d58cb14b5e99947a2f17"} Sep 10 16:13:30.318517 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.318489 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc" Sep 10 16:13:30.348413 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.348388 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9" Sep 10 16:13:30.371543 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.371511 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b5aaf3b-819c-4e40-8068-0d93b420ed05-util\") pod \"6b5aaf3b-819c-4e40-8068-0d93b420ed05\" (UID: \"6b5aaf3b-819c-4e40-8068-0d93b420ed05\") " Sep 10 16:13:30.371543 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.371542 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b5aaf3b-819c-4e40-8068-0d93b420ed05-bundle\") pod \"6b5aaf3b-819c-4e40-8068-0d93b420ed05\" (UID: \"6b5aaf3b-819c-4e40-8068-0d93b420ed05\") " Sep 10 16:13:30.371796 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.371685 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twtl6\" (UniqueName: \"kubernetes.io/projected/6b5aaf3b-819c-4e40-8068-0d93b420ed05-kube-api-access-twtl6\") pod \"6b5aaf3b-819c-4e40-8068-0d93b420ed05\" (UID: \"6b5aaf3b-819c-4e40-8068-0d93b420ed05\") " Sep 10 16:13:30.372137 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.372108 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b5aaf3b-819c-4e40-8068-0d93b420ed05-bundle" (OuterVolumeSpecName: "bundle") pod "6b5aaf3b-819c-4e40-8068-0d93b420ed05" (UID: "6b5aaf3b-819c-4e40-8068-0d93b420ed05"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:13:30.374420 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.374391 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b5aaf3b-819c-4e40-8068-0d93b420ed05-kube-api-access-twtl6" (OuterVolumeSpecName: "kube-api-access-twtl6") pod "6b5aaf3b-819c-4e40-8068-0d93b420ed05" (UID: "6b5aaf3b-819c-4e40-8068-0d93b420ed05"). InnerVolumeSpecName "kube-api-access-twtl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:13:30.376603 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.376578 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b5aaf3b-819c-4e40-8068-0d93b420ed05-util" (OuterVolumeSpecName: "util") pod "6b5aaf3b-819c-4e40-8068-0d93b420ed05" (UID: "6b5aaf3b-819c-4e40-8068-0d93b420ed05"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:13:30.472791 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.472674 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlqrc\" (UniqueName: \"kubernetes.io/projected/41fd568a-c802-4edf-9ee8-66b427a77df2-kube-api-access-qlqrc\") pod \"41fd568a-c802-4edf-9ee8-66b427a77df2\" (UID: \"41fd568a-c802-4edf-9ee8-66b427a77df2\") " Sep 10 16:13:30.472791 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.472746 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/41fd568a-c802-4edf-9ee8-66b427a77df2-bundle\") pod \"41fd568a-c802-4edf-9ee8-66b427a77df2\" (UID: \"41fd568a-c802-4edf-9ee8-66b427a77df2\") " Sep 10 16:13:30.472791 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.472780 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/41fd568a-c802-4edf-9ee8-66b427a77df2-util\") pod \"41fd568a-c802-4edf-9ee8-66b427a77df2\" (UID: \"41fd568a-c802-4edf-9ee8-66b427a77df2\") " Sep 10 16:13:30.473079 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.473027 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twtl6\" (UniqueName: \"kubernetes.io/projected/6b5aaf3b-819c-4e40-8068-0d93b420ed05-kube-api-access-twtl6\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:13:30.473079 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.473045 2541 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b5aaf3b-819c-4e40-8068-0d93b420ed05-util\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:13:30.473079 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.473058 2541 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b5aaf3b-819c-4e40-8068-0d93b420ed05-bundle\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:13:30.473292 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.473251 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fd568a-c802-4edf-9ee8-66b427a77df2-bundle" (OuterVolumeSpecName: "bundle") pod "41fd568a-c802-4edf-9ee8-66b427a77df2" (UID: "41fd568a-c802-4edf-9ee8-66b427a77df2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:13:30.475029 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.475004 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41fd568a-c802-4edf-9ee8-66b427a77df2-kube-api-access-qlqrc" (OuterVolumeSpecName: "kube-api-access-qlqrc") pod "41fd568a-c802-4edf-9ee8-66b427a77df2" (UID: "41fd568a-c802-4edf-9ee8-66b427a77df2"). InnerVolumeSpecName "kube-api-access-qlqrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:13:30.478232 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.478196 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fd568a-c802-4edf-9ee8-66b427a77df2-util" (OuterVolumeSpecName: "util") pod "41fd568a-c802-4edf-9ee8-66b427a77df2" (UID: "41fd568a-c802-4edf-9ee8-66b427a77df2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:13:30.573650 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.573596 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlqrc\" (UniqueName: \"kubernetes.io/projected/41fd568a-c802-4edf-9ee8-66b427a77df2-kube-api-access-qlqrc\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:13:30.573650 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.573644 2541 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/41fd568a-c802-4edf-9ee8-66b427a77df2-bundle\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:13:30.573650 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:30.573656 2541 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/41fd568a-c802-4edf-9ee8-66b427a77df2-util\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:13:31.184578 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.184554 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc" Sep 10 16:13:31.185004 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.184551 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9m7vc" event={"ID":"6b5aaf3b-819c-4e40-8068-0d93b420ed05","Type":"ContainerDied","Data":"c575c576d2adc0807b83975bbdf03f3b1a54960a90f5ba5a8bcc21930994118e"} Sep 10 16:13:31.185004 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.184688 2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c575c576d2adc0807b83975bbdf03f3b1a54960a90f5ba5a8bcc21930994118e" Sep 10 16:13:31.186611 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.186585 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9" event={"ID":"41fd568a-c802-4edf-9ee8-66b427a77df2","Type":"ContainerDied","Data":"e595ba4ed05e3e84a74403bd40fa91541e3bb0c82118346b72c24b6c4775289b"} Sep 10 16:13:31.186772 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.186627 2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e595ba4ed05e3e84a74403bd40fa91541e3bb0c82118346b72c24b6c4775289b" Sep 10 16:13:31.187360 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.186992 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d768jw9" Sep 10 16:13:31.330891 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.330869 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9" Sep 10 16:13:31.341700 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.341671 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw" Sep 10 16:13:31.380847 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.380824 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/465a84a6-6f3f-4c01-8311-2d56c1ea4611-bundle\") pod \"465a84a6-6f3f-4c01-8311-2d56c1ea4611\" (UID: \"465a84a6-6f3f-4c01-8311-2d56c1ea4611\") " Sep 10 16:13:31.381000 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.380938 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/465a84a6-6f3f-4c01-8311-2d56c1ea4611-util\") pod \"465a84a6-6f3f-4c01-8311-2d56c1ea4611\" (UID: \"465a84a6-6f3f-4c01-8311-2d56c1ea4611\") " Sep 10 16:13:31.381000 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.380973 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mb5g\" (UniqueName: \"kubernetes.io/projected/465a84a6-6f3f-4c01-8311-2d56c1ea4611-kube-api-access-4mb5g\") pod \"465a84a6-6f3f-4c01-8311-2d56c1ea4611\" (UID: \"465a84a6-6f3f-4c01-8311-2d56c1ea4611\") " Sep 10 16:13:31.381271 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.381251 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465a84a6-6f3f-4c01-8311-2d56c1ea4611-bundle" (OuterVolumeSpecName: "bundle") pod "465a84a6-6f3f-4c01-8311-2d56c1ea4611" (UID: "465a84a6-6f3f-4c01-8311-2d56c1ea4611"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:13:31.383220 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.383194 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465a84a6-6f3f-4c01-8311-2d56c1ea4611-kube-api-access-4mb5g" (OuterVolumeSpecName: "kube-api-access-4mb5g") pod "465a84a6-6f3f-4c01-8311-2d56c1ea4611" (UID: "465a84a6-6f3f-4c01-8311-2d56c1ea4611"). InnerVolumeSpecName "kube-api-access-4mb5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:13:31.390505 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.390466 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465a84a6-6f3f-4c01-8311-2d56c1ea4611-util" (OuterVolumeSpecName: "util") pod "465a84a6-6f3f-4c01-8311-2d56c1ea4611" (UID: "465a84a6-6f3f-4c01-8311-2d56c1ea4611"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:13:31.481594 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.481557 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lzr9\" (UniqueName: \"kubernetes.io/projected/77faf9bc-856f-40a9-8633-31b88cc8d8bf-kube-api-access-5lzr9\") pod \"77faf9bc-856f-40a9-8633-31b88cc8d8bf\" (UID: \"77faf9bc-856f-40a9-8633-31b88cc8d8bf\") " Sep 10 16:13:31.481594 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.481595 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77faf9bc-856f-40a9-8633-31b88cc8d8bf-util\") pod \"77faf9bc-856f-40a9-8633-31b88cc8d8bf\" (UID: \"77faf9bc-856f-40a9-8633-31b88cc8d8bf\") " Sep 10 16:13:31.481853 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.481646 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77faf9bc-856f-40a9-8633-31b88cc8d8bf-bundle\") pod \"77faf9bc-856f-40a9-8633-31b88cc8d8bf\" (UID: \"77faf9bc-856f-40a9-8633-31b88cc8d8bf\") " Sep 10 16:13:31.481853 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.481823 2541 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/465a84a6-6f3f-4c01-8311-2d56c1ea4611-util\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:13:31.481853 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.481841 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mb5g\" (UniqueName: \"kubernetes.io/projected/465a84a6-6f3f-4c01-8311-2d56c1ea4611-kube-api-access-4mb5g\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:13:31.482009 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.481855 2541 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/465a84a6-6f3f-4c01-8311-2d56c1ea4611-bundle\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:13:31.482340 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.482289 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77faf9bc-856f-40a9-8633-31b88cc8d8bf-bundle" (OuterVolumeSpecName: "bundle") pod "77faf9bc-856f-40a9-8633-31b88cc8d8bf" (UID: "77faf9bc-856f-40a9-8633-31b88cc8d8bf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:13:31.483941 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.483918 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77faf9bc-856f-40a9-8633-31b88cc8d8bf-kube-api-access-5lzr9" (OuterVolumeSpecName: "kube-api-access-5lzr9") pod "77faf9bc-856f-40a9-8633-31b88cc8d8bf" (UID: "77faf9bc-856f-40a9-8633-31b88cc8d8bf"). InnerVolumeSpecName "kube-api-access-5lzr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:13:31.486939 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.486906 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77faf9bc-856f-40a9-8633-31b88cc8d8bf-util" (OuterVolumeSpecName: "util") pod "77faf9bc-856f-40a9-8633-31b88cc8d8bf" (UID: "77faf9bc-856f-40a9-8633-31b88cc8d8bf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:13:31.582907 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.582818 2541 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77faf9bc-856f-40a9-8633-31b88cc8d8bf-bundle\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:13:31.582907 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.582850 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lzr9\" (UniqueName: \"kubernetes.io/projected/77faf9bc-856f-40a9-8633-31b88cc8d8bf-kube-api-access-5lzr9\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:13:31.582907 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.582864 2541 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77faf9bc-856f-40a9-8633-31b88cc8d8bf-util\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:13:31.940155 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:31.940123 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6787b8b9fd-ghlgz"] Sep 10 16:13:32.194047 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:32.193945 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw" event={"ID":"77faf9bc-856f-40a9-8633-31b88cc8d8bf","Type":"ContainerDied","Data":"03103ee686297dbcb71227efe976d31504b116b2b64bd9738b14f7ba722a8aea"} Sep 10 16:13:32.194047 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:32.194002 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7ftw" Sep 10 16:13:32.194549 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:32.194009 2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03103ee686297dbcb71227efe976d31504b116b2b64bd9738b14f7ba722a8aea" Sep 10 16:13:32.194719 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:32.194699 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:32.196005 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:32.195985 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9" event={"ID":"465a84a6-6f3f-4c01-8311-2d56c1ea4611","Type":"ContainerDied","Data":"b2718ef86aa34a209e03bd931b295ae33bd862bd75775e73c96ea50de8430596"} Sep 10 16:13:32.196112 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:32.196016 2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2718ef86aa34a209e03bd931b295ae33bd862bd75775e73c96ea50de8430596" Sep 10 16:13:32.196112 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:32.196025 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pbp9" Sep 10 16:13:32.269995 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:32.269909 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:36.540310 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:36.540276 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:36.937429 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:36.937401 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:37.695773 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:37.695717 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:40.680707 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.680668 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj"] Sep 10 16:13:40.681058 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681027 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41fd568a-c802-4edf-9ee8-66b427a77df2" containerName="pull" Sep 10 16:13:40.681058 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681038 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="41fd568a-c802-4edf-9ee8-66b427a77df2" containerName="pull" Sep 10 16:13:40.681058 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681047 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b5aaf3b-819c-4e40-8068-0d93b420ed05" containerName="pull" Sep 10 16:13:40.681058 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681053 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b5aaf3b-819c-4e40-8068-0d93b420ed05" containerName="pull" Sep 10 16:13:40.681058 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681060 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77faf9bc-856f-40a9-8633-31b88cc8d8bf" containerName="pull" Sep 10 16:13:40.681228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681065 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="77faf9bc-856f-40a9-8633-31b88cc8d8bf" containerName="pull" Sep 10 16:13:40.681228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681073 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77faf9bc-856f-40a9-8633-31b88cc8d8bf" containerName="util" Sep 10 16:13:40.681228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681078 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="77faf9bc-856f-40a9-8633-31b88cc8d8bf" containerName="util" Sep 10 16:13:40.681228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681087 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41fd568a-c802-4edf-9ee8-66b427a77df2" containerName="util" Sep 10 16:13:40.681228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681096 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="41fd568a-c802-4edf-9ee8-66b427a77df2" containerName="util" Sep 10 16:13:40.681228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681106 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77faf9bc-856f-40a9-8633-31b88cc8d8bf" containerName="extract" Sep 10 16:13:40.681228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681112 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="77faf9bc-856f-40a9-8633-31b88cc8d8bf" containerName="extract" Sep 10 16:13:40.681228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681122 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b5aaf3b-819c-4e40-8068-0d93b420ed05" containerName="util" Sep 10 16:13:40.681228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681127 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b5aaf3b-819c-4e40-8068-0d93b420ed05" containerName="util" Sep 10 16:13:40.681228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681135 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b5aaf3b-819c-4e40-8068-0d93b420ed05" containerName="extract" Sep 10 16:13:40.681228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681139 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b5aaf3b-819c-4e40-8068-0d93b420ed05" containerName="extract" Sep 10 16:13:40.681228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681146 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="465a84a6-6f3f-4c01-8311-2d56c1ea4611" containerName="pull" Sep 10 16:13:40.681228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681150 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="465a84a6-6f3f-4c01-8311-2d56c1ea4611" containerName="pull" Sep 10 16:13:40.681228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681164 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="465a84a6-6f3f-4c01-8311-2d56c1ea4611" containerName="util" Sep 10 16:13:40.681228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681173 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="465a84a6-6f3f-4c01-8311-2d56c1ea4611" containerName="util" Sep 10 16:13:40.681228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681182 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41fd568a-c802-4edf-9ee8-66b427a77df2" containerName="extract" Sep 10 16:13:40.681228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681188 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="41fd568a-c802-4edf-9ee8-66b427a77df2" containerName="extract" Sep 10 16:13:40.681228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681193 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="465a84a6-6f3f-4c01-8311-2d56c1ea4611" containerName="extract" Sep 10 16:13:40.681228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681198 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="465a84a6-6f3f-4c01-8311-2d56c1ea4611" containerName="extract" Sep 10 16:13:40.681803 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681251 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b5aaf3b-819c-4e40-8068-0d93b420ed05" containerName="extract" Sep 10 16:13:40.681803 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681265 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="77faf9bc-856f-40a9-8633-31b88cc8d8bf" containerName="extract" Sep 10 16:13:40.681803 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681272 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="41fd568a-c802-4edf-9ee8-66b427a77df2" containerName="extract" Sep 10 16:13:40.681803 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.681279 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="465a84a6-6f3f-4c01-8311-2d56c1ea4611" containerName="extract" Sep 10 16:13:40.684397 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.684381 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" Sep 10 16:13:40.686973 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.686948 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n4rkc\"" Sep 10 16:13:40.687087 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.686948 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 10 16:13:40.687150 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.687134 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 10 16:13:40.700548 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.700524 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj"] Sep 10 16:13:40.761468 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.761427 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr7b2\" (UniqueName: \"kubernetes.io/projected/0dd0169d-ed93-4040-8f79-8662189ba9bd-kube-api-access-sr7b2\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5rklj\" (UID: \"0dd0169d-ed93-4040-8f79-8662189ba9bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" Sep 10 16:13:40.761666 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.761523 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0dd0169d-ed93-4040-8f79-8662189ba9bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5rklj\" (UID: \"0dd0169d-ed93-4040-8f79-8662189ba9bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" Sep 10 16:13:40.862629 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.862591 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0dd0169d-ed93-4040-8f79-8662189ba9bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5rklj\" (UID: \"0dd0169d-ed93-4040-8f79-8662189ba9bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" Sep 10 16:13:40.862808 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.862675 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr7b2\" (UniqueName: \"kubernetes.io/projected/0dd0169d-ed93-4040-8f79-8662189ba9bd-kube-api-access-sr7b2\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5rklj\" (UID: \"0dd0169d-ed93-4040-8f79-8662189ba9bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" Sep 10 16:13:40.862995 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.862975 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0dd0169d-ed93-4040-8f79-8662189ba9bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5rklj\" (UID: \"0dd0169d-ed93-4040-8f79-8662189ba9bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" Sep 10 16:13:40.870002 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.869974 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:40.873413 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.873382 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr7b2\" (UniqueName: \"kubernetes.io/projected/0dd0169d-ed93-4040-8f79-8662189ba9bd-kube-api-access-sr7b2\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5rklj\" (UID: \"0dd0169d-ed93-4040-8f79-8662189ba9bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" Sep 10 16:13:40.996760 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:40.996688 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n4rkc\"" Sep 10 16:13:41.004629 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:41.004611 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" Sep 10 16:13:41.019647 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:41.019606 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:41.300201 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:41.300179 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj"] Sep 10 16:13:41.302889 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:13:41.302852 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd0169d_ed93_4040_8f79_8662189ba9bd.slice/crio-237e084244a9aadfb237101f194bfe4c1a2742c63d83f57fa8e2526ef726f483 WatchSource:0}: Error finding container 237e084244a9aadfb237101f194bfe4c1a2742c63d83f57fa8e2526ef726f483: Status 404 returned error can't find the container with id 237e084244a9aadfb237101f194bfe4c1a2742c63d83f57fa8e2526ef726f483 Sep 10 16:13:41.329665 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:41.329630 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:42.249424 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:42.249385 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" event={"ID":"0dd0169d-ed93-4040-8f79-8662189ba9bd","Type":"ContainerStarted","Data":"237e084244a9aadfb237101f194bfe4c1a2742c63d83f57fa8e2526ef726f483"} Sep 10 16:13:43.162382 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:43.162203 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:43.451501 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:43.451408 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:43.572992 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:43.572948 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:46.026662 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:46.026634 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:46.278552 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:46.278448 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" event={"ID":"0dd0169d-ed93-4040-8f79-8662189ba9bd","Type":"ContainerStarted","Data":"19dcf0296696d298451542a4e62b6bc52688d4fe34062716d462842e48ed0dd0"} Sep 10 16:13:46.278552 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:46.278528 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" Sep 10 16:13:46.349440 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:46.349385 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" podStartSLOduration=2.213337009 podStartE2EDuration="6.349369261s" totalImagesPullingTime="4.136032252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:13:46.342713531 +0000 UTC m=+725.865958728" watchObservedRunningTime="2026-09-10 16:13:46.349369261 +0000 UTC m=+725.872614457" Sep 10 16:13:46.405701 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:46.405673 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:46.541478 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:46.541400 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:47.992046 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:47.992014 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4"] Sep 10 16:13:47.995543 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:47.995526 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4" Sep 10 16:13:48.001233 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:48.001214 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-2mb7p\"" Sep 10 16:13:48.024504 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:48.024479 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4"] Sep 10 16:13:48.138325 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:48.138255 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26brp\" (UniqueName: \"kubernetes.io/projected/75cf486f-3390-425f-8aed-c46c1d98a7da-kube-api-access-26brp\") pod \"limitador-operator-controller-manager-74495c69f8-5gfb4\" (UID: \"75cf486f-3390-425f-8aed-c46c1d98a7da\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4" Sep 10 16:13:48.239918 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:48.239885 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26brp\" (UniqueName: \"kubernetes.io/projected/75cf486f-3390-425f-8aed-c46c1d98a7da-kube-api-access-26brp\") pod \"limitador-operator-controller-manager-74495c69f8-5gfb4\" (UID: \"75cf486f-3390-425f-8aed-c46c1d98a7da\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4" Sep 10 16:13:48.258914 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:48.258850 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26brp\" (UniqueName: \"kubernetes.io/projected/75cf486f-3390-425f-8aed-c46c1d98a7da-kube-api-access-26brp\") pod \"limitador-operator-controller-manager-74495c69f8-5gfb4\" (UID: \"75cf486f-3390-425f-8aed-c46c1d98a7da\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4" Sep 10 16:13:48.301421 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:48.301393 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:48.305241 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:48.305222 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4" Sep 10 16:13:48.594594 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:48.594514 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:48.610809 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:48.610784 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4"] Sep 10 16:13:48.612165 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:13:48.612140 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75cf486f_3390_425f_8aed_c46c1d98a7da.slice/crio-f74ee2eb51e02d4e70404f517a59f3019a08dc0e0283b6140edaf6cd60e6e37b WatchSource:0}: Error finding container f74ee2eb51e02d4e70404f517a59f3019a08dc0e0283b6140edaf6cd60e6e37b: Status 404 returned error can't find the container with id f74ee2eb51e02d4e70404f517a59f3019a08dc0e0283b6140edaf6cd60e6e37b Sep 10 16:13:49.017223 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:49.017193 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:49.296069 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:49.295988 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4" event={"ID":"75cf486f-3390-425f-8aed-c46c1d98a7da","Type":"ContainerStarted","Data":"f74ee2eb51e02d4e70404f517a59f3019a08dc0e0283b6140edaf6cd60e6e37b"} Sep 10 16:13:49.855240 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:49.855210 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:51.309376 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:51.309338 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4" event={"ID":"75cf486f-3390-425f-8aed-c46c1d98a7da","Type":"ContainerStarted","Data":"62d91d1400c9264fb40f1044e4bb0383dad57f8c9ad1a59813a448a00feebf70"} Sep 10 16:13:51.309848 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:51.309436 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4" Sep 10 16:13:51.343343 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:51.343273 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4" podStartSLOduration=2.423855091 podStartE2EDuration="4.343257542s" totalImagesPullingTime="1.919402451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:13:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:13:51.340583229 +0000 UTC m=+730.863828426" watchObservedRunningTime="2026-09-10 16:13:51.343257542 +0000 UTC m=+730.866502739" Sep 10 16:13:51.540118 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:51.540082 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:51.637898 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:51.637870 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:51.960586 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:51.960505 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:52.062464 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:52.062425 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:52.362943 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:52.362850 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:52.611960 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:52.611907 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:54.398064 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:54.398026 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:56.960722 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:56.960681 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6787b8b9fd-ghlgz" podUID="312fe479-86ab-456a-89f2-7a5480a95bae" containerName="console" containerID="cri-o://9d9ce547dacffbddba9320afb0d8c5b33a2bec8e54c9c3e3750e1dec6b2dca38" gracePeriod=15 Sep 10 16:13:57.044650 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.044617 2541 patch_prober.go:28] interesting pod/console-6787b8b9fd-ghlgz container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.132.0.24:8443/health\": dial tcp 10.132.0.24:8443: connect: connection refused" start-of-body= Sep 10 16:13:57.044827 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.044669 2541 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-6787b8b9fd-ghlgz" podUID="312fe479-86ab-456a-89f2-7a5480a95bae" containerName="console" probeResult="failure" output="Get \"https://10.132.0.24:8443/health\": dial tcp 10.132.0.24:8443: connect: connection refused" Sep 10 16:13:57.212691 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.212628 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:13:57.216197 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.216175 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/312fe479-86ab-456a-89f2-7a5480a95bae-console-oauth-config\") pod \"312fe479-86ab-456a-89f2-7a5480a95bae\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " Sep 10 16:13:57.216358 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.216258 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-trusted-ca-bundle\") pod \"312fe479-86ab-456a-89f2-7a5480a95bae\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " Sep 10 16:13:57.216358 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.216276 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-oauth-serving-cert\") pod \"312fe479-86ab-456a-89f2-7a5480a95bae\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " Sep 10 16:13:57.216358 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.216292 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-console-config\") pod \"312fe479-86ab-456a-89f2-7a5480a95bae\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " Sep 10 16:13:57.216358 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.216329 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-service-ca\") pod \"312fe479-86ab-456a-89f2-7a5480a95bae\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " Sep 10 16:13:57.216583 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.216371 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/312fe479-86ab-456a-89f2-7a5480a95bae-console-serving-cert\") pod \"312fe479-86ab-456a-89f2-7a5480a95bae\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " Sep 10 16:13:57.216583 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.216402 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fjq4\" (UniqueName: \"kubernetes.io/projected/312fe479-86ab-456a-89f2-7a5480a95bae-kube-api-access-2fjq4\") pod \"312fe479-86ab-456a-89f2-7a5480a95bae\" (UID: \"312fe479-86ab-456a-89f2-7a5480a95bae\") " Sep 10 16:13:57.216686 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.216662 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "312fe479-86ab-456a-89f2-7a5480a95bae" (UID: "312fe479-86ab-456a-89f2-7a5480a95bae"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:13:57.216739 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.216683 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "312fe479-86ab-456a-89f2-7a5480a95bae" (UID: "312fe479-86ab-456a-89f2-7a5480a95bae"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:13:57.216739 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.216706 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-console-config" (OuterVolumeSpecName: "console-config") pod "312fe479-86ab-456a-89f2-7a5480a95bae" (UID: "312fe479-86ab-456a-89f2-7a5480a95bae"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:13:57.216844 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.216747 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-service-ca" (OuterVolumeSpecName: "service-ca") pod "312fe479-86ab-456a-89f2-7a5480a95bae" (UID: "312fe479-86ab-456a-89f2-7a5480a95bae"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:13:57.218495 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.218471 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/312fe479-86ab-456a-89f2-7a5480a95bae-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "312fe479-86ab-456a-89f2-7a5480a95bae" (UID: "312fe479-86ab-456a-89f2-7a5480a95bae"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:13:57.218662 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.218644 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/312fe479-86ab-456a-89f2-7a5480a95bae-kube-api-access-2fjq4" (OuterVolumeSpecName: "kube-api-access-2fjq4") pod "312fe479-86ab-456a-89f2-7a5480a95bae" (UID: "312fe479-86ab-456a-89f2-7a5480a95bae"). InnerVolumeSpecName "kube-api-access-2fjq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:13:57.218773 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.218754 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/312fe479-86ab-456a-89f2-7a5480a95bae-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "312fe479-86ab-456a-89f2-7a5480a95bae" (UID: "312fe479-86ab-456a-89f2-7a5480a95bae"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:13:57.286470 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.286441 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" Sep 10 16:13:57.317002 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.316969 2541 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-trusted-ca-bundle\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:13:57.317002 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.316992 2541 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-oauth-serving-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:13:57.317002 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.317003 2541 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-console-config\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:13:57.317226 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.317014 2541 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/312fe479-86ab-456a-89f2-7a5480a95bae-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:13:57.317226 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.317027 2541 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/312fe479-86ab-456a-89f2-7a5480a95bae-console-serving-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:13:57.317226 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.317040 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fjq4\" (UniqueName: \"kubernetes.io/projected/312fe479-86ab-456a-89f2-7a5480a95bae-kube-api-access-2fjq4\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:13:57.317226 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.317049 2541 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/312fe479-86ab-456a-89f2-7a5480a95bae-console-oauth-config\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:13:57.342987 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.342952 2541 generic.go:356] "Generic (PLEG): container finished" podID="312fe479-86ab-456a-89f2-7a5480a95bae" containerID="9d9ce547dacffbddba9320afb0d8c5b33a2bec8e54c9c3e3750e1dec6b2dca38" exitCode=0 Sep 10 16:13:57.343169 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.343009 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6787b8b9fd-ghlgz" Sep 10 16:13:57.343169 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.343032 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6787b8b9fd-ghlgz" event={"ID":"312fe479-86ab-456a-89f2-7a5480a95bae","Type":"ContainerDied","Data":"9d9ce547dacffbddba9320afb0d8c5b33a2bec8e54c9c3e3750e1dec6b2dca38"} Sep 10 16:13:57.343169 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.343084 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6787b8b9fd-ghlgz" event={"ID":"312fe479-86ab-456a-89f2-7a5480a95bae","Type":"ContainerDied","Data":"0ce32e2173c8608cda4bbc470bf902bd592048e853bc88b9c9f3310ff8639891"} Sep 10 16:13:57.343169 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.343109 2541 scope.go:117] "RemoveContainer" containerID="9d9ce547dacffbddba9320afb0d8c5b33a2bec8e54c9c3e3750e1dec6b2dca38" Sep 10 16:13:57.353775 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.353654 2541 scope.go:117] "RemoveContainer" containerID="9d9ce547dacffbddba9320afb0d8c5b33a2bec8e54c9c3e3750e1dec6b2dca38" Sep 10 16:13:57.354409 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:13:57.354384 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d9ce547dacffbddba9320afb0d8c5b33a2bec8e54c9c3e3750e1dec6b2dca38\": container with ID starting with 9d9ce547dacffbddba9320afb0d8c5b33a2bec8e54c9c3e3750e1dec6b2dca38 not found: ID does not exist" containerID="9d9ce547dacffbddba9320afb0d8c5b33a2bec8e54c9c3e3750e1dec6b2dca38" Sep 10 16:13:57.354522 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.354420 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9ce547dacffbddba9320afb0d8c5b33a2bec8e54c9c3e3750e1dec6b2dca38"} err="failed to get container status \"9d9ce547dacffbddba9320afb0d8c5b33a2bec8e54c9c3e3750e1dec6b2dca38\": rpc error: code = NotFound desc = could not find container \"9d9ce547dacffbddba9320afb0d8c5b33a2bec8e54c9c3e3750e1dec6b2dca38\": container with ID starting with 9d9ce547dacffbddba9320afb0d8c5b33a2bec8e54c9c3e3750e1dec6b2dca38 not found: ID does not exist" Sep 10 16:13:57.378562 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.378532 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6787b8b9fd-ghlgz"] Sep 10 16:13:57.383245 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:57.383216 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6787b8b9fd-ghlgz"] Sep 10 16:13:58.979805 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:58.979770 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="312fe479-86ab-456a-89f2-7a5480a95bae" path="/var/lib/kubelet/pods/312fe479-86ab-456a-89f2-7a5480a95bae/volumes" Sep 10 16:13:59.072978 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:59.072945 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:59.175545 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:13:59.175514 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:01.290151 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.290115 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj"] Sep 10 16:14:01.290613 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.290323 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" podUID="0dd0169d-ed93-4040-8f79-8662189ba9bd" containerName="manager" containerID="cri-o://19dcf0296696d298451542a4e62b6bc52688d4fe34062716d462842e48ed0dd0" gracePeriod=2 Sep 10 16:14:01.295208 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.295185 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj"] Sep 10 16:14:01.351397 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.351367 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4"] Sep 10 16:14:01.351620 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.351590 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4" podUID="75cf486f-3390-425f-8aed-c46c1d98a7da" containerName="manager" containerID="cri-o://62d91d1400c9264fb40f1044e4bb0383dad57f8c9ad1a59813a448a00feebf70" gracePeriod=2 Sep 10 16:14:01.354421 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.354184 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4" Sep 10 16:14:01.355150 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.354868 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4"] Sep 10 16:14:01.375118 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.375093 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h"] Sep 10 16:14:01.375467 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.375438 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dd0169d-ed93-4040-8f79-8662189ba9bd" containerName="manager" Sep 10 16:14:01.375467 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.375453 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dd0169d-ed93-4040-8f79-8662189ba9bd" containerName="manager" Sep 10 16:14:01.375467 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.375466 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="312fe479-86ab-456a-89f2-7a5480a95bae" containerName="console" Sep 10 16:14:01.375467 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.375471 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="312fe479-86ab-456a-89f2-7a5480a95bae" containerName="console" Sep 10 16:14:01.375719 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.375486 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75cf486f-3390-425f-8aed-c46c1d98a7da" containerName="manager" Sep 10 16:14:01.375719 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.375491 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="75cf486f-3390-425f-8aed-c46c1d98a7da" containerName="manager" Sep 10 16:14:01.375719 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.375539 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="312fe479-86ab-456a-89f2-7a5480a95bae" containerName="console" Sep 10 16:14:01.375719 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.375546 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dd0169d-ed93-4040-8f79-8662189ba9bd" containerName="manager" Sep 10 16:14:01.375719 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.375555 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="75cf486f-3390-425f-8aed-c46c1d98a7da" containerName="manager" Sep 10 16:14:01.378400 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.378382 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h" Sep 10 16:14:01.390073 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:14:01.390034 2541 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5rklj\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="0dd0169d-ed93-4040-8f79-8662189ba9bd" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" Sep 10 16:14:01.393794 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:14:01.393756 2541 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5gfb4\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="75cf486f-3390-425f-8aed-c46c1d98a7da" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4" Sep 10 16:14:01.395402 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:14:01.395363 2541 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5rklj\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="0dd0169d-ed93-4040-8f79-8662189ba9bd" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" Sep 10 16:14:01.396046 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.396028 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ftxxj"] Sep 10 16:14:01.396696 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:14:01.396668 2541 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5gfb4\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="75cf486f-3390-425f-8aed-c46c1d98a7da" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4" Sep 10 16:14:01.399379 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.399359 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ftxxj" Sep 10 16:14:01.446593 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.446565 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjkks\" (UniqueName: \"kubernetes.io/projected/540fb80f-556f-4e15-b04d-dadddd67d31d-kube-api-access-gjkks\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6jt8h\" (UID: \"540fb80f-556f-4e15-b04d-dadddd67d31d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h" Sep 10 16:14:01.446741 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.446601 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/540fb80f-556f-4e15-b04d-dadddd67d31d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6jt8h\" (UID: \"540fb80f-556f-4e15-b04d-dadddd67d31d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h" Sep 10 16:14:01.446741 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.446625 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgw76\" (UniqueName: \"kubernetes.io/projected/a519db3c-1725-4d67-82fc-f88e519425aa-kube-api-access-kgw76\") pod \"limitador-operator-controller-manager-74495c69f8-ftxxj\" (UID: \"a519db3c-1725-4d67-82fc-f88e519425aa\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ftxxj" Sep 10 16:14:01.459950 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.459833 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ftxxj"] Sep 10 16:14:01.472595 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.472564 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h"] Sep 10 16:14:01.501472 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:14:01.501409 2541 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5gfb4\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="75cf486f-3390-425f-8aed-c46c1d98a7da" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4" Sep 10 16:14:01.503922 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:14:01.503893 2541 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5rklj\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="0dd0169d-ed93-4040-8f79-8662189ba9bd" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" Sep 10 16:14:01.547619 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.547554 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjkks\" (UniqueName: \"kubernetes.io/projected/540fb80f-556f-4e15-b04d-dadddd67d31d-kube-api-access-gjkks\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6jt8h\" (UID: \"540fb80f-556f-4e15-b04d-dadddd67d31d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h" Sep 10 16:14:01.547619 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.547602 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/540fb80f-556f-4e15-b04d-dadddd67d31d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6jt8h\" (UID: \"540fb80f-556f-4e15-b04d-dadddd67d31d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h" Sep 10 16:14:01.547782 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.547640 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgw76\" (UniqueName: \"kubernetes.io/projected/a519db3c-1725-4d67-82fc-f88e519425aa-kube-api-access-kgw76\") pod \"limitador-operator-controller-manager-74495c69f8-ftxxj\" (UID: \"a519db3c-1725-4d67-82fc-f88e519425aa\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ftxxj" Sep 10 16:14:01.547997 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.547978 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/540fb80f-556f-4e15-b04d-dadddd67d31d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6jt8h\" (UID: \"540fb80f-556f-4e15-b04d-dadddd67d31d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h" Sep 10 16:14:01.568231 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.568201 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgw76\" (UniqueName: \"kubernetes.io/projected/a519db3c-1725-4d67-82fc-f88e519425aa-kube-api-access-kgw76\") pod \"limitador-operator-controller-manager-74495c69f8-ftxxj\" (UID: \"a519db3c-1725-4d67-82fc-f88e519425aa\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ftxxj" Sep 10 16:14:01.568514 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.568492 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjkks\" (UniqueName: \"kubernetes.io/projected/540fb80f-556f-4e15-b04d-dadddd67d31d-kube-api-access-gjkks\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6jt8h\" (UID: \"540fb80f-556f-4e15-b04d-dadddd67d31d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h" Sep 10 16:14:01.598601 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.598580 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" Sep 10 16:14:01.600699 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.600681 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4" Sep 10 16:14:01.601559 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:14:01.601539 2541 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5rklj\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="0dd0169d-ed93-4040-8f79-8662189ba9bd" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" Sep 10 16:14:01.604766 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:14:01.604747 2541 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5gfb4\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="75cf486f-3390-425f-8aed-c46c1d98a7da" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4" Sep 10 16:14:01.606392 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:14:01.606377 2541 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5rklj\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="0dd0169d-ed93-4040-8f79-8662189ba9bd" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" Sep 10 16:14:01.611796 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:14:01.611777 2541 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5gfb4\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="75cf486f-3390-425f-8aed-c46c1d98a7da" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4" Sep 10 16:14:01.648172 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.648142 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26brp\" (UniqueName: \"kubernetes.io/projected/75cf486f-3390-425f-8aed-c46c1d98a7da-kube-api-access-26brp\") pod \"75cf486f-3390-425f-8aed-c46c1d98a7da\" (UID: \"75cf486f-3390-425f-8aed-c46c1d98a7da\") " Sep 10 16:14:01.648323 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.648188 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0dd0169d-ed93-4040-8f79-8662189ba9bd-extensions-socket-volume\") pod \"0dd0169d-ed93-4040-8f79-8662189ba9bd\" (UID: \"0dd0169d-ed93-4040-8f79-8662189ba9bd\") " Sep 10 16:14:01.648323 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.648222 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sr7b2\" (UniqueName: \"kubernetes.io/projected/0dd0169d-ed93-4040-8f79-8662189ba9bd-kube-api-access-sr7b2\") pod \"0dd0169d-ed93-4040-8f79-8662189ba9bd\" (UID: \"0dd0169d-ed93-4040-8f79-8662189ba9bd\") " Sep 10 16:14:01.648746 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.648718 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd0169d-ed93-4040-8f79-8662189ba9bd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0dd0169d-ed93-4040-8f79-8662189ba9bd" (UID: "0dd0169d-ed93-4040-8f79-8662189ba9bd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:14:01.650605 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.650578 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75cf486f-3390-425f-8aed-c46c1d98a7da-kube-api-access-26brp" (OuterVolumeSpecName: "kube-api-access-26brp") pod "75cf486f-3390-425f-8aed-c46c1d98a7da" (UID: "75cf486f-3390-425f-8aed-c46c1d98a7da"). InnerVolumeSpecName "kube-api-access-26brp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:14:01.650676 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.650640 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd0169d-ed93-4040-8f79-8662189ba9bd-kube-api-access-sr7b2" (OuterVolumeSpecName: "kube-api-access-sr7b2") pod "0dd0169d-ed93-4040-8f79-8662189ba9bd" (UID: "0dd0169d-ed93-4040-8f79-8662189ba9bd"). InnerVolumeSpecName "kube-api-access-sr7b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:14:01.733373 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.733344 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h" Sep 10 16:14:01.738418 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.738398 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ftxxj" Sep 10 16:14:01.749661 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.749640 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26brp\" (UniqueName: \"kubernetes.io/projected/75cf486f-3390-425f-8aed-c46c1d98a7da-kube-api-access-26brp\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:14:01.749748 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.749664 2541 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0dd0169d-ed93-4040-8f79-8662189ba9bd-extensions-socket-volume\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:14:01.749748 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:01.749675 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr7b2\" (UniqueName: \"kubernetes.io/projected/0dd0169d-ed93-4040-8f79-8662189ba9bd-kube-api-access-sr7b2\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:14:02.033635 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:14:02.033605 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda519db3c_1725_4d67_82fc_f88e519425aa.slice/crio-b6c4c55d8fc238c78acf075046bf51ad5c59b9ad9ac2c460643439aba7d3b63c WatchSource:0}: Error finding container b6c4c55d8fc238c78acf075046bf51ad5c59b9ad9ac2c460643439aba7d3b63c: Status 404 returned error can't find the container with id b6c4c55d8fc238c78acf075046bf51ad5c59b9ad9ac2c460643439aba7d3b63c Sep 10 16:14:02.033791 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:02.033770 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ftxxj"] Sep 10 16:14:02.038648 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:02.038627 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h"] Sep 10 16:14:02.039065 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:14:02.039044 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540fb80f_556f_4e15_b04d_dadddd67d31d.slice/crio-39e69d898921228e3523bd7d5b274776c54f809435bf9ee8acb76f0833821b8b WatchSource:0}: Error finding container 39e69d898921228e3523bd7d5b274776c54f809435bf9ee8acb76f0833821b8b: Status 404 returned error can't find the container with id 39e69d898921228e3523bd7d5b274776c54f809435bf9ee8acb76f0833821b8b Sep 10 16:14:02.374045 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:02.374012 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ftxxj" event={"ID":"a519db3c-1725-4d67-82fc-f88e519425aa","Type":"ContainerStarted","Data":"4aec6841f654a1b368f761d732949a382dca1ccddcbf14198e2432e4e3b0e46e"} Sep 10 16:14:02.374486 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:02.374161 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ftxxj" Sep 10 16:14:02.374486 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:02.374219 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ftxxj" event={"ID":"a519db3c-1725-4d67-82fc-f88e519425aa","Type":"ContainerStarted","Data":"b6c4c55d8fc238c78acf075046bf51ad5c59b9ad9ac2c460643439aba7d3b63c"} Sep 10 16:14:02.375703 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:02.375683 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h" event={"ID":"540fb80f-556f-4e15-b04d-dadddd67d31d","Type":"ContainerStarted","Data":"6e174efd9a4ae3cd8da1e884c0fd490cc80b1ff5d7e437389a404709fd50c2aa"} Sep 10 16:14:02.375808 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:02.375720 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h" event={"ID":"540fb80f-556f-4e15-b04d-dadddd67d31d","Type":"ContainerStarted","Data":"39e69d898921228e3523bd7d5b274776c54f809435bf9ee8acb76f0833821b8b"} Sep 10 16:14:02.375808 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:02.375785 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h" Sep 10 16:14:02.376444 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:14:02.376420 2541 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5rklj\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="0dd0169d-ed93-4040-8f79-8662189ba9bd" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" Sep 10 16:14:02.377057 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:02.377033 2541 generic.go:356] "Generic (PLEG): container finished" podID="75cf486f-3390-425f-8aed-c46c1d98a7da" containerID="62d91d1400c9264fb40f1044e4bb0383dad57f8c9ad1a59813a448a00feebf70" exitCode=0 Sep 10 16:14:02.377135 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:02.377073 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4" Sep 10 16:14:02.377202 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:02.377072 2541 scope.go:117] "RemoveContainer" containerID="62d91d1400c9264fb40f1044e4bb0383dad57f8c9ad1a59813a448a00feebf70" Sep 10 16:14:02.379324 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:02.378638 2541 generic.go:356] "Generic (PLEG): container finished" podID="0dd0169d-ed93-4040-8f79-8662189ba9bd" containerID="19dcf0296696d298451542a4e62b6bc52688d4fe34062716d462842e48ed0dd0" exitCode=0 Sep 10 16:14:02.379324 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:02.378684 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" Sep 10 16:14:02.393542 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:02.393520 2541 scope.go:117] "RemoveContainer" containerID="62d91d1400c9264fb40f1044e4bb0383dad57f8c9ad1a59813a448a00feebf70" Sep 10 16:14:02.393880 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:14:02.393858 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62d91d1400c9264fb40f1044e4bb0383dad57f8c9ad1a59813a448a00feebf70\": container with ID starting with 62d91d1400c9264fb40f1044e4bb0383dad57f8c9ad1a59813a448a00feebf70 not found: ID does not exist" containerID="62d91d1400c9264fb40f1044e4bb0383dad57f8c9ad1a59813a448a00feebf70" Sep 10 16:14:02.393972 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:02.393887 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d91d1400c9264fb40f1044e4bb0383dad57f8c9ad1a59813a448a00feebf70"} err="failed to get container status \"62d91d1400c9264fb40f1044e4bb0383dad57f8c9ad1a59813a448a00feebf70\": rpc error: code = NotFound desc = could not find container \"62d91d1400c9264fb40f1044e4bb0383dad57f8c9ad1a59813a448a00feebf70\": container with ID starting with 62d91d1400c9264fb40f1044e4bb0383dad57f8c9ad1a59813a448a00feebf70 not found: ID does not exist" Sep 10 16:14:02.393972 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:02.393905 2541 scope.go:117] "RemoveContainer" containerID="19dcf0296696d298451542a4e62b6bc52688d4fe34062716d462842e48ed0dd0" Sep 10 16:14:02.403570 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:02.403550 2541 scope.go:117] "RemoveContainer" containerID="19dcf0296696d298451542a4e62b6bc52688d4fe34062716d462842e48ed0dd0" Sep 10 16:14:02.403850 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:14:02.403831 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19dcf0296696d298451542a4e62b6bc52688d4fe34062716d462842e48ed0dd0\": container with ID starting with 19dcf0296696d298451542a4e62b6bc52688d4fe34062716d462842e48ed0dd0 not found: ID does not exist" containerID="19dcf0296696d298451542a4e62b6bc52688d4fe34062716d462842e48ed0dd0" Sep 10 16:14:02.403895 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:02.403858 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19dcf0296696d298451542a4e62b6bc52688d4fe34062716d462842e48ed0dd0"} err="failed to get container status \"19dcf0296696d298451542a4e62b6bc52688d4fe34062716d462842e48ed0dd0\": rpc error: code = NotFound desc = could not find container \"19dcf0296696d298451542a4e62b6bc52688d4fe34062716d462842e48ed0dd0\": container with ID starting with 19dcf0296696d298451542a4e62b6bc52688d4fe34062716d462842e48ed0dd0 not found: ID does not exist" Sep 10 16:14:02.433549 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:02.433493 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ftxxj" podStartSLOduration=1.433478866 podStartE2EDuration="1.433478866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:14:02.432718064 +0000 UTC m=+741.955963262" watchObservedRunningTime="2026-09-10 16:14:02.433478866 +0000 UTC m=+741.956724053" Sep 10 16:14:02.451280 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:14:02.451247 2541 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5gfb4\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="75cf486f-3390-425f-8aed-c46c1d98a7da" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4" Sep 10 16:14:02.456256 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:14:02.456227 2541 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5gfb4\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="75cf486f-3390-425f-8aed-c46c1d98a7da" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5gfb4" Sep 10 16:14:02.518315 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:02.518244 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h" podStartSLOduration=1.518225758 podStartE2EDuration="1.518225758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:14:02.516715907 +0000 UTC m=+742.039961104" watchObservedRunningTime="2026-09-10 16:14:02.518225758 +0000 UTC m=+742.041470955" Sep 10 16:14:02.518660 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:14:02.518632 2541 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5rklj\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="0dd0169d-ed93-4040-8f79-8662189ba9bd" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rklj" Sep 10 16:14:02.980558 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:02.980515 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dd0169d-ed93-4040-8f79-8662189ba9bd" path="/var/lib/kubelet/pods/0dd0169d-ed93-4040-8f79-8662189ba9bd/volumes" Sep 10 16:14:02.981026 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:02.981008 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75cf486f-3390-425f-8aed-c46c1d98a7da" path="/var/lib/kubelet/pods/75cf486f-3390-425f-8aed-c46c1d98a7da/volumes" Sep 10 16:14:04.595143 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:04.595107 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:04.714748 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:04.714714 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:08.497908 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:08.497874 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:08.598736 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:08.598703 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:12.871805 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:12.871773 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:13.026060 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:13.026028 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:13.170128 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:13.170053 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:13.388653 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:13.388626 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ftxxj" Sep 10 16:14:13.388818 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:13.388699 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h" Sep 10 16:14:17.147490 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.147456 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h"] Sep 10 16:14:17.147948 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.147659 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h" podUID="540fb80f-556f-4e15-b04d-dadddd67d31d" containerName="manager" containerID="cri-o://6e174efd9a4ae3cd8da1e884c0fd490cc80b1ff5d7e437389a404709fd50c2aa" gracePeriod=10 Sep 10 16:14:17.397248 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.397223 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h" Sep 10 16:14:17.464125 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.464041 2541 generic.go:356] "Generic (PLEG): container finished" podID="540fb80f-556f-4e15-b04d-dadddd67d31d" containerID="6e174efd9a4ae3cd8da1e884c0fd490cc80b1ff5d7e437389a404709fd50c2aa" exitCode=0 Sep 10 16:14:17.464125 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.464089 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h" Sep 10 16:14:17.464125 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.464101 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h" event={"ID":"540fb80f-556f-4e15-b04d-dadddd67d31d","Type":"ContainerDied","Data":"6e174efd9a4ae3cd8da1e884c0fd490cc80b1ff5d7e437389a404709fd50c2aa"} Sep 10 16:14:17.464421 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.464149 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h" event={"ID":"540fb80f-556f-4e15-b04d-dadddd67d31d","Type":"ContainerDied","Data":"39e69d898921228e3523bd7d5b274776c54f809435bf9ee8acb76f0833821b8b"} Sep 10 16:14:17.464421 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.464170 2541 scope.go:117] "RemoveContainer" containerID="6e174efd9a4ae3cd8da1e884c0fd490cc80b1ff5d7e437389a404709fd50c2aa" Sep 10 16:14:17.472624 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.472608 2541 scope.go:117] "RemoveContainer" containerID="6e174efd9a4ae3cd8da1e884c0fd490cc80b1ff5d7e437389a404709fd50c2aa" Sep 10 16:14:17.472908 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:14:17.472883 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e174efd9a4ae3cd8da1e884c0fd490cc80b1ff5d7e437389a404709fd50c2aa\": container with ID starting with 6e174efd9a4ae3cd8da1e884c0fd490cc80b1ff5d7e437389a404709fd50c2aa not found: ID does not exist" containerID="6e174efd9a4ae3cd8da1e884c0fd490cc80b1ff5d7e437389a404709fd50c2aa" Sep 10 16:14:17.472956 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.472917 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e174efd9a4ae3cd8da1e884c0fd490cc80b1ff5d7e437389a404709fd50c2aa"} err="failed to get container status \"6e174efd9a4ae3cd8da1e884c0fd490cc80b1ff5d7e437389a404709fd50c2aa\": rpc error: code = NotFound desc = could not find container \"6e174efd9a4ae3cd8da1e884c0fd490cc80b1ff5d7e437389a404709fd50c2aa\": container with ID starting with 6e174efd9a4ae3cd8da1e884c0fd490cc80b1ff5d7e437389a404709fd50c2aa not found: ID does not exist" Sep 10 16:14:17.480225 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.480202 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjkks\" (UniqueName: \"kubernetes.io/projected/540fb80f-556f-4e15-b04d-dadddd67d31d-kube-api-access-gjkks\") pod \"540fb80f-556f-4e15-b04d-dadddd67d31d\" (UID: \"540fb80f-556f-4e15-b04d-dadddd67d31d\") " Sep 10 16:14:17.480314 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.480265 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/540fb80f-556f-4e15-b04d-dadddd67d31d-extensions-socket-volume\") pod \"540fb80f-556f-4e15-b04d-dadddd67d31d\" (UID: \"540fb80f-556f-4e15-b04d-dadddd67d31d\") " Sep 10 16:14:17.480647 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.480628 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/540fb80f-556f-4e15-b04d-dadddd67d31d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "540fb80f-556f-4e15-b04d-dadddd67d31d" (UID: "540fb80f-556f-4e15-b04d-dadddd67d31d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:14:17.482346 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.482328 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540fb80f-556f-4e15-b04d-dadddd67d31d-kube-api-access-gjkks" (OuterVolumeSpecName: "kube-api-access-gjkks") pod "540fb80f-556f-4e15-b04d-dadddd67d31d" (UID: "540fb80f-556f-4e15-b04d-dadddd67d31d"). InnerVolumeSpecName "kube-api-access-gjkks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:14:17.516326 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.516286 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6"] Sep 10 16:14:17.516687 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.516675 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="540fb80f-556f-4e15-b04d-dadddd67d31d" containerName="manager" Sep 10 16:14:17.516733 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.516690 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="540fb80f-556f-4e15-b04d-dadddd67d31d" containerName="manager" Sep 10 16:14:17.516768 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.516759 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="540fb80f-556f-4e15-b04d-dadddd67d31d" containerName="manager" Sep 10 16:14:17.520801 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.520783 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6" Sep 10 16:14:17.535948 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.535885 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6"] Sep 10 16:14:17.581635 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.581609 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8d2baaf-45b6-4250-9b68-6653285e8951-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-jwzb6\" (UID: \"b8d2baaf-45b6-4250-9b68-6653285e8951\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6" Sep 10 16:14:17.581785 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.581660 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbnxx\" (UniqueName: \"kubernetes.io/projected/b8d2baaf-45b6-4250-9b68-6653285e8951-kube-api-access-gbnxx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-jwzb6\" (UID: \"b8d2baaf-45b6-4250-9b68-6653285e8951\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6" Sep 10 16:14:17.581785 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.581735 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjkks\" (UniqueName: \"kubernetes.io/projected/540fb80f-556f-4e15-b04d-dadddd67d31d-kube-api-access-gjkks\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:14:17.581785 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.581753 2541 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/540fb80f-556f-4e15-b04d-dadddd67d31d-extensions-socket-volume\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:14:17.682932 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.682897 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8d2baaf-45b6-4250-9b68-6653285e8951-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-jwzb6\" (UID: \"b8d2baaf-45b6-4250-9b68-6653285e8951\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6" Sep 10 16:14:17.683093 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.682955 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbnxx\" (UniqueName: \"kubernetes.io/projected/b8d2baaf-45b6-4250-9b68-6653285e8951-kube-api-access-gbnxx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-jwzb6\" (UID: \"b8d2baaf-45b6-4250-9b68-6653285e8951\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6" Sep 10 16:14:17.683253 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.683233 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8d2baaf-45b6-4250-9b68-6653285e8951-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-jwzb6\" (UID: \"b8d2baaf-45b6-4250-9b68-6653285e8951\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6" Sep 10 16:14:17.701984 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.701961 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbnxx\" (UniqueName: \"kubernetes.io/projected/b8d2baaf-45b6-4250-9b68-6653285e8951-kube-api-access-gbnxx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-jwzb6\" (UID: \"b8d2baaf-45b6-4250-9b68-6653285e8951\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6" Sep 10 16:14:17.800770 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.800736 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h"] Sep 10 16:14:17.802821 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.802796 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6jt8h"] Sep 10 16:14:17.831328 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:17.831285 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6" Sep 10 16:14:18.122893 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:18.122869 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6"] Sep 10 16:14:18.123910 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:14:18.123886 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d2baaf_45b6_4250_9b68_6653285e8951.slice/crio-ff8e1d6a7d4ffb3c57cce959bd718a05ffea47c31a002a1efe0a7b12ec89beb4 WatchSource:0}: Error finding container ff8e1d6a7d4ffb3c57cce959bd718a05ffea47c31a002a1efe0a7b12ec89beb4: Status 404 returned error can't find the container with id ff8e1d6a7d4ffb3c57cce959bd718a05ffea47c31a002a1efe0a7b12ec89beb4 Sep 10 16:14:18.470724 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:18.470685 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6" event={"ID":"b8d2baaf-45b6-4250-9b68-6653285e8951","Type":"ContainerStarted","Data":"0d62b080aeaeb9a8c28395021e1f093a5c146ba61272ac47ef75ea71f44e50aa"} Sep 10 16:14:18.471137 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:18.470788 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6" event={"ID":"b8d2baaf-45b6-4250-9b68-6653285e8951","Type":"ContainerStarted","Data":"ff8e1d6a7d4ffb3c57cce959bd718a05ffea47c31a002a1efe0a7b12ec89beb4"} Sep 10 16:14:18.471137 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:18.470906 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6" Sep 10 16:14:18.494294 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:18.494248 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6" podStartSLOduration=1.494233009 podStartE2EDuration="1.494233009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:14:18.492099852 +0000 UTC m=+758.015345049" watchObservedRunningTime="2026-09-10 16:14:18.494233009 +0000 UTC m=+758.017478206" Sep 10 16:14:18.979927 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:18.979896 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="540fb80f-556f-4e15-b04d-dadddd67d31d" path="/var/lib/kubelet/pods/540fb80f-556f-4e15-b04d-dadddd67d31d/volumes" Sep 10 16:14:19.611636 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:19.611600 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:19.754114 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:19.754077 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:26.015677 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:26.015637 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:26.149877 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:26.149839 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:26.268357 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:26.268243 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:28.951315 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:28.951280 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:29.268674 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:29.268588 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:29.479920 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:29.479891 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6" Sep 10 16:14:34.025264 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:34.025230 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:34.140864 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:34.140829 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:34.240768 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:34.240730 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:34.541193 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:34.541163 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:34.642813 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:34.642782 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:34.759108 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:34.759078 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:34.889514 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:34.889480 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:41.757259 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:41.757225 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:41.861945 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:41.861901 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:47.033469 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:47.033431 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:47.217809 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:47.217779 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:47.623539 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:47.623506 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:47.813058 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:47.813021 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:14:47.819669 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:47.819639 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-gzlkl" Sep 10 16:14:47.822408 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:47.822384 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 10 16:14:47.822547 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:47.822442 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qlltw\"" Sep 10 16:14:47.829877 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:47.829855 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:14:47.870623 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:47.870584 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:14:47.952858 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:47.952785 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c143d4ff-7d79-41b8-8898-dea4091300e1-config-file\") pod \"limitador-limitador-84994f7ffc-gzlkl\" (UID: \"c143d4ff-7d79-41b8-8898-dea4091300e1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gzlkl" Sep 10 16:14:47.952858 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:47.952842 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqdfh\" (UniqueName: \"kubernetes.io/projected/c143d4ff-7d79-41b8-8898-dea4091300e1-kube-api-access-wqdfh\") pod \"limitador-limitador-84994f7ffc-gzlkl\" (UID: \"c143d4ff-7d79-41b8-8898-dea4091300e1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gzlkl" Sep 10 16:14:47.984360 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:47.984325 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:48.054341 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:48.054287 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c143d4ff-7d79-41b8-8898-dea4091300e1-config-file\") pod \"limitador-limitador-84994f7ffc-gzlkl\" (UID: \"c143d4ff-7d79-41b8-8898-dea4091300e1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gzlkl" Sep 10 16:14:48.054738 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:48.054409 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqdfh\" (UniqueName: \"kubernetes.io/projected/c143d4ff-7d79-41b8-8898-dea4091300e1-kube-api-access-wqdfh\") pod \"limitador-limitador-84994f7ffc-gzlkl\" (UID: \"c143d4ff-7d79-41b8-8898-dea4091300e1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gzlkl" Sep 10 16:14:48.054978 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:48.054961 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c143d4ff-7d79-41b8-8898-dea4091300e1-config-file\") pod \"limitador-limitador-84994f7ffc-gzlkl\" (UID: \"c143d4ff-7d79-41b8-8898-dea4091300e1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gzlkl" Sep 10 16:14:48.066778 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:48.066754 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqdfh\" (UniqueName: \"kubernetes.io/projected/c143d4ff-7d79-41b8-8898-dea4091300e1-kube-api-access-wqdfh\") pod \"limitador-limitador-84994f7ffc-gzlkl\" (UID: \"c143d4ff-7d79-41b8-8898-dea4091300e1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gzlkl" Sep 10 16:14:48.092977 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:48.092943 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:48.131136 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:48.131099 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-gzlkl" Sep 10 16:14:48.198650 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:48.198518 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:48.410269 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:48.410242 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:14:48.412002 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:14:48.411976 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc143d4ff_7d79_41b8_8898_dea4091300e1.slice/crio-38ac1beb8bb8d06d5b40c796f75adcc9db69986154ab8cd652fb027d7745c00e WatchSource:0}: Error finding container 38ac1beb8bb8d06d5b40c796f75adcc9db69986154ab8cd652fb027d7745c00e: Status 404 returned error can't find the container with id 38ac1beb8bb8d06d5b40c796f75adcc9db69986154ab8cd652fb027d7745c00e Sep 10 16:14:48.568108 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:48.568075 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:48.639313 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:48.639265 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-gzlkl" event={"ID":"c143d4ff-7d79-41b8-8898-dea4091300e1","Type":"ContainerStarted","Data":"38ac1beb8bb8d06d5b40c796f75adcc9db69986154ab8cd652fb027d7745c00e"} Sep 10 16:14:48.660706 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:48.660620 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:49.645972 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:49.645931 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-gzlkl" event={"ID":"c143d4ff-7d79-41b8-8898-dea4091300e1","Type":"ContainerStarted","Data":"cfb9b6c6b1790d181ee5e5a82933fc5b91e83a07f76477ffb95546f96325f694"} Sep 10 16:14:49.646473 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:49.646045 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-gzlkl" Sep 10 16:14:49.666462 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:49.666370 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-gzlkl" podStartSLOduration=1.677239245 podStartE2EDuration="2.666355054s" totalImagesPullingTime="989.115809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:14:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:14:49.664432911 +0000 UTC m=+789.187678107" watchObservedRunningTime="2026-09-10 16:14:49.666355054 +0000 UTC m=+789.189600286" Sep 10 16:14:52.796004 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:52.795972 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:52.895473 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:52.895437 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:57.467251 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:14:57.467210 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:15:00.653034 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:15:00.653000 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-gzlkl" Sep 10 16:15:02.839265 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:15:02.839189 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:15:02.990865 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:15:02.990834 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:15:03.276994 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:15:03.276961 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:15:15.754834 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:15:15.754797 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:15:16.173137 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:15:16.173107 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:15:16.862476 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:15:16.862437 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:15:17.405508 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:15:17.405479 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:15:25.431209 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:15:25.431176 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:15:25.530605 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:15:25.530572 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:15:25.632474 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:15:25.632440 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:15:25.732632 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:15:25.732547 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:15:28.838170 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:15:28.838134 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:15:28.941195 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:15:28.941145 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:16:00.165993 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:00.165955 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:16:02.190420 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:02.190389 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:16:02.315509 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:02.315475 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:16:02.525238 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:02.525153 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:16:02.685586 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:02.685552 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:16:02.686001 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:02.685983 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-85d2w"] Sep 10 16:16:02.689431 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:02.689416 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-85d2w" Sep 10 16:16:02.691468 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:02.691451 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 10 16:16:02.691561 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:02.691517 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-fnm9l\"" Sep 10 16:16:02.702729 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:02.702707 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-85d2w"] Sep 10 16:16:02.810035 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:02.809948 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n742m\" (UniqueName: \"kubernetes.io/projected/c9c7e7f6-7f76-493c-aa7a-f69a70592836-kube-api-access-n742m\") pod \"payload-processing-5b54b7fbb7-85d2w\" (UID: \"c9c7e7f6-7f76-493c-aa7a-f69a70592836\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-85d2w" Sep 10 16:16:02.810035 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:02.810012 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c7e7f6-7f76-493c-aa7a-f69a70592836-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-85d2w\" (UID: \"c9c7e7f6-7f76-493c-aa7a-f69a70592836\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-85d2w" Sep 10 16:16:02.816162 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:02.816117 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:16:02.911428 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:02.911379 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c7e7f6-7f76-493c-aa7a-f69a70592836-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-85d2w\" (UID: \"c9c7e7f6-7f76-493c-aa7a-f69a70592836\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-85d2w" Sep 10 16:16:02.911633 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:02.911498 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n742m\" (UniqueName: \"kubernetes.io/projected/c9c7e7f6-7f76-493c-aa7a-f69a70592836-kube-api-access-n742m\") pod \"payload-processing-5b54b7fbb7-85d2w\" (UID: \"c9c7e7f6-7f76-493c-aa7a-f69a70592836\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-85d2w" Sep 10 16:16:02.912179 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:02.912156 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c7e7f6-7f76-493c-aa7a-f69a70592836-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-85d2w\" (UID: \"c9c7e7f6-7f76-493c-aa7a-f69a70592836\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-85d2w" Sep 10 16:16:02.922433 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:02.922414 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n742m\" (UniqueName: \"kubernetes.io/projected/c9c7e7f6-7f76-493c-aa7a-f69a70592836-kube-api-access-n742m\") pod \"payload-processing-5b54b7fbb7-85d2w\" (UID: \"c9c7e7f6-7f76-493c-aa7a-f69a70592836\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-85d2w" Sep 10 16:16:02.999528 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:02.999493 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-85d2w" Sep 10 16:16:03.145393 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:03.145361 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:16:03.287609 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:03.287515 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-85d2w"] Sep 10 16:16:03.290261 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:16:03.290236 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c7e7f6_7f76_493c_aa7a_f69a70592836.slice/crio-decabdd6f2a8676d81ba644a402f99daa06e051805cdc08f24c05e17285e5385 WatchSource:0}: Error finding container decabdd6f2a8676d81ba644a402f99daa06e051805cdc08f24c05e17285e5385: Status 404 returned error can't find the container with id decabdd6f2a8676d81ba644a402f99daa06e051805cdc08f24c05e17285e5385 Sep 10 16:16:03.292040 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:03.292025 2541 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 16:16:04.036322 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:04.036274 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-85d2w" event={"ID":"c9c7e7f6-7f76-493c-aa7a-f69a70592836","Type":"ContainerStarted","Data":"decabdd6f2a8676d81ba644a402f99daa06e051805cdc08f24c05e17285e5385"} Sep 10 16:16:05.045514 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:05.045479 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-85d2w" event={"ID":"c9c7e7f6-7f76-493c-aa7a-f69a70592836","Type":"ContainerStarted","Data":"0519057d99b07574c42e9986ff875f6b70e8ddfda5899b102276901c171e0482"} Sep 10 16:16:05.045955 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:05.045643 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-85d2w" Sep 10 16:16:05.073623 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:05.073569 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-85d2w" podStartSLOduration=1.456093267 podStartE2EDuration="3.07355351s" totalImagesPullingTime="1.617460243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:16:05.072335613 +0000 UTC m=+864.595580808" watchObservedRunningTime="2026-09-10 16:16:05.07355351 +0000 UTC m=+864.596798698" Sep 10 16:16:12.635645 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:12.635610 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:16:12.743551 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:12.743514 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:16:16.053709 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:16.053673 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-85d2w" Sep 10 16:16:23.057674 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:23.057628 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:16:23.762745 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:23.762708 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:16:36.876505 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:36.876422 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll"] Sep 10 16:16:36.880056 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:36.880035 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:36.889104 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:36.889078 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 10 16:16:36.891522 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:36.891504 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 16:16:36.893483 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:36.893468 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wz694\"" Sep 10 16:16:36.894964 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:36.894947 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 16:16:36.908724 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:36.908693 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g5qz\" (UniqueName: \"kubernetes.io/projected/2b159143-5244-4115-80c0-79b2e9828784-kube-api-access-7g5qz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cd2ll\" (UID: \"2b159143-5244-4115-80c0-79b2e9828784\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:36.908830 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:36.908759 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b159143-5244-4115-80c0-79b2e9828784-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cd2ll\" (UID: \"2b159143-5244-4115-80c0-79b2e9828784\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:36.908830 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:36.908801 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b159143-5244-4115-80c0-79b2e9828784-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cd2ll\" (UID: \"2b159143-5244-4115-80c0-79b2e9828784\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:36.908923 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:36.908833 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b159143-5244-4115-80c0-79b2e9828784-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cd2ll\" (UID: \"2b159143-5244-4115-80c0-79b2e9828784\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:36.908923 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:36.908863 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b159143-5244-4115-80c0-79b2e9828784-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cd2ll\" (UID: \"2b159143-5244-4115-80c0-79b2e9828784\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:36.908923 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:36.908896 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b159143-5244-4115-80c0-79b2e9828784-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cd2ll\" (UID: \"2b159143-5244-4115-80c0-79b2e9828784\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:36.935758 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:36.935735 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:16:37.009787 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:37.009756 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g5qz\" (UniqueName: \"kubernetes.io/projected/2b159143-5244-4115-80c0-79b2e9828784-kube-api-access-7g5qz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cd2ll\" (UID: \"2b159143-5244-4115-80c0-79b2e9828784\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:37.009958 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:37.009798 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b159143-5244-4115-80c0-79b2e9828784-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cd2ll\" (UID: \"2b159143-5244-4115-80c0-79b2e9828784\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:37.009958 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:37.009846 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b159143-5244-4115-80c0-79b2e9828784-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cd2ll\" (UID: \"2b159143-5244-4115-80c0-79b2e9828784\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:37.009958 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:37.009874 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b159143-5244-4115-80c0-79b2e9828784-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cd2ll\" (UID: \"2b159143-5244-4115-80c0-79b2e9828784\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:37.009958 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:37.009906 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b159143-5244-4115-80c0-79b2e9828784-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cd2ll\" (UID: \"2b159143-5244-4115-80c0-79b2e9828784\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:37.009958 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:37.009936 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b159143-5244-4115-80c0-79b2e9828784-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cd2ll\" (UID: \"2b159143-5244-4115-80c0-79b2e9828784\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:37.010244 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:37.010221 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b159143-5244-4115-80c0-79b2e9828784-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cd2ll\" (UID: \"2b159143-5244-4115-80c0-79b2e9828784\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:37.010343 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:37.010277 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b159143-5244-4115-80c0-79b2e9828784-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cd2ll\" (UID: \"2b159143-5244-4115-80c0-79b2e9828784\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:37.010343 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:37.010335 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b159143-5244-4115-80c0-79b2e9828784-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cd2ll\" (UID: \"2b159143-5244-4115-80c0-79b2e9828784\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:37.012277 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:37.012253 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b159143-5244-4115-80c0-79b2e9828784-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cd2ll\" (UID: \"2b159143-5244-4115-80c0-79b2e9828784\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:37.012569 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:37.012552 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b159143-5244-4115-80c0-79b2e9828784-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cd2ll\" (UID: \"2b159143-5244-4115-80c0-79b2e9828784\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:37.024269 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:37.024248 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g5qz\" (UniqueName: \"kubernetes.io/projected/2b159143-5244-4115-80c0-79b2e9828784-kube-api-access-7g5qz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cd2ll\" (UID: \"2b159143-5244-4115-80c0-79b2e9828784\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:37.029254 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:37.029229 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll"] Sep 10 16:16:37.207400 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:37.207308 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:37.491827 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:37.491790 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll"] Sep 10 16:16:37.492355 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:16:37.492329 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b159143_5244_4115_80c0_79b2e9828784.slice/crio-53cdcfc6974da930e75dec9d7e4e9e492da2ce0b921a81230fda3526fb6df0e5 WatchSource:0}: Error finding container 53cdcfc6974da930e75dec9d7e4e9e492da2ce0b921a81230fda3526fb6df0e5: Status 404 returned error can't find the container with id 53cdcfc6974da930e75dec9d7e4e9e492da2ce0b921a81230fda3526fb6df0e5 Sep 10 16:16:37.673075 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:37.673038 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:16:38.232573 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:38.232524 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" event={"ID":"2b159143-5244-4115-80c0-79b2e9828784","Type":"ContainerStarted","Data":"53cdcfc6974da930e75dec9d7e4e9e492da2ce0b921a81230fda3526fb6df0e5"} Sep 10 16:16:39.240626 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:39.240585 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" event={"ID":"2b159143-5244-4115-80c0-79b2e9828784","Type":"ContainerStarted","Data":"5895aea27518d2b1edc66053a6a8ecc8a24dbc81b546a4b4fe85ed65eb3fd667"} Sep 10 16:16:39.275652 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:39.275593 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" podStartSLOduration=2.242149763 podStartE2EDuration="3.275575842s" totalImagesPullingTime="1.033426079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:16:39.272291148 +0000 UTC m=+898.795536345" watchObservedRunningTime="2026-09-10 16:16:39.275575842 +0000 UTC m=+898.798821078" Sep 10 16:16:40.990521 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:40.990482 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/1.log" Sep 10 16:16:40.990521 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:40.990520 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/1.log" Sep 10 16:16:40.996410 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:40.996391 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4c6j_2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd/ovn-acl-logging/0.log" Sep 10 16:16:40.996587 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:40.996569 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4c6j_2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd/ovn-acl-logging/0.log" Sep 10 16:16:47.207894 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:47.207849 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:47.208434 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:47.207924 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:47.220570 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:47.220543 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:47.294796 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:47.294769 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cd2ll" Sep 10 16:16:47.600274 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:47.600140 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:16:48.567237 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:48.567199 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:16:50.708044 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:50.708003 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:16:51.766642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:51.766607 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:16:57.417450 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:57.417412 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:16:59.053639 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:16:59.053601 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:17:02.860531 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:02.860500 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl"] Sep 10 16:17:02.864693 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:02.864675 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:02.867215 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:02.867198 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 10 16:17:02.904559 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:02.904518 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:02.917819 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:02.917787 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl"] Sep 10 16:17:02.939420 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:02.939389 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5b90458-3710-4957-8785-0300678d368a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqxcl\" (UID: \"c5b90458-3710-4957-8785-0300678d368a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:02.939633 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:02.939445 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5b90458-3710-4957-8785-0300678d368a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqxcl\" (UID: \"c5b90458-3710-4957-8785-0300678d368a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:02.939633 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:02.939535 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5b90458-3710-4957-8785-0300678d368a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqxcl\" (UID: \"c5b90458-3710-4957-8785-0300678d368a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:02.939633 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:02.939596 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlcgw\" (UniqueName: \"kubernetes.io/projected/c5b90458-3710-4957-8785-0300678d368a-kube-api-access-zlcgw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqxcl\" (UID: \"c5b90458-3710-4957-8785-0300678d368a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:02.939633 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:02.939627 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5b90458-3710-4957-8785-0300678d368a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqxcl\" (UID: \"c5b90458-3710-4957-8785-0300678d368a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:02.939816 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:02.939683 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5b90458-3710-4957-8785-0300678d368a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqxcl\" (UID: \"c5b90458-3710-4957-8785-0300678d368a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:03.040345 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:03.040294 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5b90458-3710-4957-8785-0300678d368a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqxcl\" (UID: \"c5b90458-3710-4957-8785-0300678d368a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:03.040540 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:03.040362 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5b90458-3710-4957-8785-0300678d368a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqxcl\" (UID: \"c5b90458-3710-4957-8785-0300678d368a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:03.040540 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:03.040386 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5b90458-3710-4957-8785-0300678d368a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqxcl\" (UID: \"c5b90458-3710-4957-8785-0300678d368a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:03.040540 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:03.040414 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5b90458-3710-4957-8785-0300678d368a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqxcl\" (UID: \"c5b90458-3710-4957-8785-0300678d368a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:03.040540 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:03.040444 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlcgw\" (UniqueName: \"kubernetes.io/projected/c5b90458-3710-4957-8785-0300678d368a-kube-api-access-zlcgw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqxcl\" (UID: \"c5b90458-3710-4957-8785-0300678d368a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:03.040540 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:03.040467 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5b90458-3710-4957-8785-0300678d368a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqxcl\" (UID: \"c5b90458-3710-4957-8785-0300678d368a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:03.040817 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:03.040777 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5b90458-3710-4957-8785-0300678d368a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqxcl\" (UID: \"c5b90458-3710-4957-8785-0300678d368a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:03.040863 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:03.040816 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5b90458-3710-4957-8785-0300678d368a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqxcl\" (UID: \"c5b90458-3710-4957-8785-0300678d368a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:03.040915 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:03.040895 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5b90458-3710-4957-8785-0300678d368a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqxcl\" (UID: \"c5b90458-3710-4957-8785-0300678d368a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:03.042771 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:03.042751 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5b90458-3710-4957-8785-0300678d368a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqxcl\" (UID: \"c5b90458-3710-4957-8785-0300678d368a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:03.043032 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:03.043015 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5b90458-3710-4957-8785-0300678d368a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqxcl\" (UID: \"c5b90458-3710-4957-8785-0300678d368a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:03.053734 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:03.053711 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlcgw\" (UniqueName: \"kubernetes.io/projected/c5b90458-3710-4957-8785-0300678d368a-kube-api-access-zlcgw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqxcl\" (UID: \"c5b90458-3710-4957-8785-0300678d368a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:03.175398 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:03.175359 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:03.455425 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:03.455399 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl"] Sep 10 16:17:03.457172 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:17:03.457146 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b90458_3710_4957_8785_0300678d368a.slice/crio-2cad5dbff6343760af7240dabf2100c36a2a4ae01ff7c9739634b04ee753a6c7 WatchSource:0}: Error finding container 2cad5dbff6343760af7240dabf2100c36a2a4ae01ff7c9739634b04ee753a6c7: Status 404 returned error can't find the container with id 2cad5dbff6343760af7240dabf2100c36a2a4ae01ff7c9739634b04ee753a6c7 Sep 10 16:17:04.385032 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:04.384994 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" event={"ID":"c5b90458-3710-4957-8785-0300678d368a","Type":"ContainerStarted","Data":"23558e81ee14c41144e422919ac96dcab08871a203dff9811e7e8c10d1efb809"} Sep 10 16:17:04.385445 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:04.385045 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" event={"ID":"c5b90458-3710-4957-8785-0300678d368a","Type":"ContainerStarted","Data":"2cad5dbff6343760af7240dabf2100c36a2a4ae01ff7c9739634b04ee753a6c7"} Sep 10 16:17:04.406224 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:04.406158 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" podStartSLOduration=2.182609312 podStartE2EDuration="2.406138423s" totalImagesPullingTime="223.529111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:04.406099794 +0000 UTC m=+923.929344991" watchObservedRunningTime="2026-09-10 16:17:04.406138423 +0000 UTC m=+923.929383620" Sep 10 16:17:05.153233 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:05.153192 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:17:07.931680 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:07.931643 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:10.858236 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:10.858202 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:17:13.176434 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:13.176381 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:13.176962 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:13.176479 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:13.189251 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:13.189221 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:17:13.453253 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:17:13.453163 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqxcl" Sep 10 16:18:32.805184 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:32.805150 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:32.895622 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:32.895593 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:33.987483 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:33.987443 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:34.128750 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:34.128710 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:50.282570 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:50.282533 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:50.397782 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:50.397712 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:50.622950 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:50.622901 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:50.894949 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:50.894835 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:51.719786 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:51.719744 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:51.826071 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:51.826022 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:52.178028 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:52.177983 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:52.258604 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:52.258568 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7474-0203ec"] Sep 10 16:18:52.262219 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:52.262199 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7474-0203ec" Sep 10 16:18:52.265131 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:52.265093 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8f29h\"" Sep 10 16:18:52.273418 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:52.273391 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7474-0203ec"] Sep 10 16:18:52.392807 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:52.392752 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wvvr\" (UniqueName: \"kubernetes.io/projected/71752fb6-bf3d-4e78-91d9-3fb22e4b1fdd-kube-api-access-6wvvr\") pod \"test-curl-7474-0203ec\" (UID: \"71752fb6-bf3d-4e78-91d9-3fb22e4b1fdd\") " pod="opendatahub/test-curl-7474-0203ec" Sep 10 16:18:52.493725 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:52.493625 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wvvr\" (UniqueName: \"kubernetes.io/projected/71752fb6-bf3d-4e78-91d9-3fb22e4b1fdd-kube-api-access-6wvvr\") pod \"test-curl-7474-0203ec\" (UID: \"71752fb6-bf3d-4e78-91d9-3fb22e4b1fdd\") " pod="opendatahub/test-curl-7474-0203ec" Sep 10 16:18:52.503189 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:52.503152 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wvvr\" (UniqueName: \"kubernetes.io/projected/71752fb6-bf3d-4e78-91d9-3fb22e4b1fdd-kube-api-access-6wvvr\") pod \"test-curl-7474-0203ec\" (UID: \"71752fb6-bf3d-4e78-91d9-3fb22e4b1fdd\") " pod="opendatahub/test-curl-7474-0203ec" Sep 10 16:18:52.572867 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:52.572833 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7474-0203ec" Sep 10 16:18:52.856830 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:52.856800 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7474-0203ec"] Sep 10 16:18:52.858441 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:18:52.858417 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71752fb6_bf3d_4e78_91d9_3fb22e4b1fdd.slice/crio-da73b75d539cb0e7ffb994f401d18ff0d86df4f2b077809054763adb9f7038ab WatchSource:0}: Error finding container da73b75d539cb0e7ffb994f401d18ff0d86df4f2b077809054763adb9f7038ab: Status 404 returned error can't find the container with id da73b75d539cb0e7ffb994f401d18ff0d86df4f2b077809054763adb9f7038ab Sep 10 16:18:52.989227 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:52.989196 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7474-0203ec" event={"ID":"71752fb6-bf3d-4e78-91d9-3fb22e4b1fdd","Type":"ContainerStarted","Data":"da73b75d539cb0e7ffb994f401d18ff0d86df4f2b077809054763adb9f7038ab"} Sep 10 16:18:53.460218 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:53.460181 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:53.571805 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:53.571766 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:53.995509 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:53.995467 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7474-0203ec" event={"ID":"71752fb6-bf3d-4e78-91d9-3fb22e4b1fdd","Type":"ContainerStarted","Data":"b6b972c8db38f2055a20f3edd5c514f9b839dc2733fed6b9999ed0bde7837459"} Sep 10 16:18:54.024892 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:54.024782 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7474-0203ec" podStartSLOduration=1.129082009 podStartE2EDuration="2.024762525s" totalImagesPullingTime="895.680516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:54.022099012 +0000 UTC m=+1033.545344210" watchObservedRunningTime="2026-09-10 16:18:54.024762525 +0000 UTC m=+1033.548007722" Sep 10 16:18:54.358365 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:54.358261 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:18:54.603564 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:54.603531 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7474-0203ec"] Sep 10 16:18:54.607767 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:54.607742 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7474-0203ec"] Sep 10 16:18:54.667779 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:54.667751 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:54.755906 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:54.755871 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:54.857037 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:54.856995 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:55.001479 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:18:55.001397 2541 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7474-0203ec" container="test-curl-7474-0203ec" Sep 10 16:18:55.267188 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:55.267110 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:55.369988 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:55.369955 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:55.538273 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:55.538189 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5"] Sep 10 16:18:55.538684 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:55.538664 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71752fb6-bf3d-4e78-91d9-3fb22e4b1fdd" containerName="test-curl-7474-0203ec" Sep 10 16:18:55.538773 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:55.538687 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="71752fb6-bf3d-4e78-91d9-3fb22e4b1fdd" containerName="test-curl-7474-0203ec" Sep 10 16:18:55.538773 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:55.538763 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="71752fb6-bf3d-4e78-91d9-3fb22e4b1fdd" containerName="test-curl-7474-0203ec" Sep 10 16:18:55.542352 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:55.542328 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5" Sep 10 16:18:55.544460 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:55.544436 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-37c805\"" Sep 10 16:18:55.544651 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:55.544477 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-37c805-dockercfg-fpgph\"" Sep 10 16:18:55.552635 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:55.552610 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5"] Sep 10 16:18:55.620273 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:55.620227 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95fb447-93f6-49ba-992c-276c2303d1f4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5\" (UID: \"c95fb447-93f6-49ba-992c-276c2303d1f4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5" Sep 10 16:18:55.620273 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:55.620273 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9b7x\" (UniqueName: \"kubernetes.io/projected/c95fb447-93f6-49ba-992c-276c2303d1f4-kube-api-access-m9b7x\") pod \"payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5\" (UID: \"c95fb447-93f6-49ba-992c-276c2303d1f4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5" Sep 10 16:18:55.721203 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:55.721165 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95fb447-93f6-49ba-992c-276c2303d1f4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5\" (UID: \"c95fb447-93f6-49ba-992c-276c2303d1f4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5" Sep 10 16:18:55.721203 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:55.721205 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9b7x\" (UniqueName: \"kubernetes.io/projected/c95fb447-93f6-49ba-992c-276c2303d1f4-kube-api-access-m9b7x\") pod \"payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5\" (UID: \"c95fb447-93f6-49ba-992c-276c2303d1f4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5" Sep 10 16:18:55.722016 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:55.721995 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95fb447-93f6-49ba-992c-276c2303d1f4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5\" (UID: \"c95fb447-93f6-49ba-992c-276c2303d1f4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5" Sep 10 16:18:55.730918 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:55.730891 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9b7x\" (UniqueName: \"kubernetes.io/projected/c95fb447-93f6-49ba-992c-276c2303d1f4-kube-api-access-m9b7x\") pod \"payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5\" (UID: \"c95fb447-93f6-49ba-992c-276c2303d1f4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5" Sep 10 16:18:55.854941 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:55.854841 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5" Sep 10 16:18:56.007069 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:56.006999 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7474-0203ec" podUID="71752fb6-bf3d-4e78-91d9-3fb22e4b1fdd" containerName="test-curl-7474-0203ec" containerID="cri-o://b6b972c8db38f2055a20f3edd5c514f9b839dc2733fed6b9999ed0bde7837459" gracePeriod=2 Sep 10 16:18:56.141265 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:56.141237 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5"] Sep 10 16:18:56.143703 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:18:56.143675 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc95fb447_93f6_49ba_992c_276c2303d1f4.slice/crio-c101c646f2db56d76a7f51534ee50b95f43fcabae2faf18c797b0286f497ad35 WatchSource:0}: Error finding container c101c646f2db56d76a7f51534ee50b95f43fcabae2faf18c797b0286f497ad35: Status 404 returned error can't find the container with id c101c646f2db56d76a7f51534ee50b95f43fcabae2faf18c797b0286f497ad35 Sep 10 16:18:57.015081 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:57.015039 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5" event={"ID":"c95fb447-93f6-49ba-992c-276c2303d1f4","Type":"ContainerStarted","Data":"2812a563f7b06b6a9baf5c553f6fa05bf05b0e34a5987879418a68a442b650c4"} Sep 10 16:18:57.015482 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:57.015101 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5" Sep 10 16:18:57.015482 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:57.015118 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5" event={"ID":"c95fb447-93f6-49ba-992c-276c2303d1f4","Type":"ContainerStarted","Data":"c101c646f2db56d76a7f51534ee50b95f43fcabae2faf18c797b0286f497ad35"} Sep 10 16:18:57.039386 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:57.039288 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5" podStartSLOduration=2.039271884 podStartE2EDuration="2.039271884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:57.036767674 +0000 UTC m=+1036.560012872" watchObservedRunningTime="2026-09-10 16:18:57.039271884 +0000 UTC m=+1036.562517081" Sep 10 16:18:58.190475 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:58.190449 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7474-0203ec" Sep 10 16:18:58.346488 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:58.346400 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wvvr\" (UniqueName: \"kubernetes.io/projected/71752fb6-bf3d-4e78-91d9-3fb22e4b1fdd-kube-api-access-6wvvr\") pod \"71752fb6-bf3d-4e78-91d9-3fb22e4b1fdd\" (UID: \"71752fb6-bf3d-4e78-91d9-3fb22e4b1fdd\") " Sep 10 16:18:58.348821 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:58.348796 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71752fb6-bf3d-4e78-91d9-3fb22e4b1fdd-kube-api-access-6wvvr" (OuterVolumeSpecName: "kube-api-access-6wvvr") pod "71752fb6-bf3d-4e78-91d9-3fb22e4b1fdd" (UID: "71752fb6-bf3d-4e78-91d9-3fb22e4b1fdd"). InnerVolumeSpecName "kube-api-access-6wvvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:18:58.447497 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:58.447457 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wvvr\" (UniqueName: \"kubernetes.io/projected/71752fb6-bf3d-4e78-91d9-3fb22e4b1fdd-kube-api-access-6wvvr\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:18:58.741708 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:58.741655 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:58.980420 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:58.980385 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71752fb6-bf3d-4e78-91d9-3fb22e4b1fdd" path="/var/lib/kubelet/pods/71752fb6-bf3d-4e78-91d9-3fb22e4b1fdd/volumes" Sep 10 16:18:59.030027 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:59.029924 2541 generic.go:356] "Generic (PLEG): container finished" podID="71752fb6-bf3d-4e78-91d9-3fb22e4b1fdd" containerID="b6b972c8db38f2055a20f3edd5c514f9b839dc2733fed6b9999ed0bde7837459" exitCode=137 Sep 10 16:18:59.030027 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:59.029976 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7474-0203ec" Sep 10 16:18:59.030027 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:59.030004 2541 scope.go:117] "RemoveContainer" containerID="b6b972c8db38f2055a20f3edd5c514f9b839dc2733fed6b9999ed0bde7837459" Sep 10 16:18:59.042332 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:59.042277 2541 scope.go:117] "RemoveContainer" containerID="b6b972c8db38f2055a20f3edd5c514f9b839dc2733fed6b9999ed0bde7837459" Sep 10 16:18:59.042660 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:18:59.042638 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6b972c8db38f2055a20f3edd5c514f9b839dc2733fed6b9999ed0bde7837459\": container with ID starting with b6b972c8db38f2055a20f3edd5c514f9b839dc2733fed6b9999ed0bde7837459 not found: ID does not exist" containerID="b6b972c8db38f2055a20f3edd5c514f9b839dc2733fed6b9999ed0bde7837459" Sep 10 16:18:59.042721 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:59.042672 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6b972c8db38f2055a20f3edd5c514f9b839dc2733fed6b9999ed0bde7837459"} err="failed to get container status \"b6b972c8db38f2055a20f3edd5c514f9b839dc2733fed6b9999ed0bde7837459\": rpc error: code = NotFound desc = could not find container \"b6b972c8db38f2055a20f3edd5c514f9b839dc2733fed6b9999ed0bde7837459\": container with ID starting with b6b972c8db38f2055a20f3edd5c514f9b839dc2733fed6b9999ed0bde7837459 not found: ID does not exist" Sep 10 16:18:59.155786 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:59.155755 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:59.582019 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:18:59.581972 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:00.255601 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:00.255570 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:00.357977 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:00.357948 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:00.510427 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:00.510334 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:00.829151 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:00.829068 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:00.945041 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:00.945008 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:01.956919 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:01.956881 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:02.054458 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:02.054420 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:02.602396 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:02.602364 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:02.703717 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:02.703678 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:02.970687 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:02.970658 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:03.085325 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:03.085262 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:03.732952 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:03.732921 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:03.833215 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:03.833182 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:04.268347 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.268286 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:04.362032 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.362003 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:04.398547 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.398511 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl"] Sep 10 16:19:04.403281 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.403261 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl" Sep 10 16:19:04.406532 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.406512 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-1596ac\"" Sep 10 16:19:04.406682 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.406632 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-1596ac-dockercfg-jhcjg\"" Sep 10 16:19:04.422866 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.422838 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl"] Sep 10 16:19:04.448910 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.448876 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4"] Sep 10 16:19:04.456081 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.454509 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4" Sep 10 16:19:04.470627 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.470599 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4"] Sep 10 16:19:04.504229 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.504194 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwbgl\" (UniqueName: \"kubernetes.io/projected/75a705b7-eccf-42e1-b238-e98a5018d2ca-kube-api-access-wwbgl\") pod \"payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl\" (UID: \"75a705b7-eccf-42e1-b238-e98a5018d2ca\") " pod="openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl" Sep 10 16:19:04.504403 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.504236 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a705b7-eccf-42e1-b238-e98a5018d2ca-plugins-config-volume\") pod \"payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl\" (UID: \"75a705b7-eccf-42e1-b238-e98a5018d2ca\") " pod="openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl" Sep 10 16:19:04.605124 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.605026 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8qhl\" (UniqueName: \"kubernetes.io/projected/3bdfa708-7fd8-44bd-a474-8726fbe75466-kube-api-access-b8qhl\") pod \"payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4\" (UID: \"3bdfa708-7fd8-44bd-a474-8726fbe75466\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4" Sep 10 16:19:04.605124 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.605077 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bdfa708-7fd8-44bd-a474-8726fbe75466-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4\" (UID: \"3bdfa708-7fd8-44bd-a474-8726fbe75466\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4" Sep 10 16:19:04.605363 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.605165 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwbgl\" (UniqueName: \"kubernetes.io/projected/75a705b7-eccf-42e1-b238-e98a5018d2ca-kube-api-access-wwbgl\") pod \"payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl\" (UID: \"75a705b7-eccf-42e1-b238-e98a5018d2ca\") " pod="openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl" Sep 10 16:19:04.605363 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.605188 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a705b7-eccf-42e1-b238-e98a5018d2ca-plugins-config-volume\") pod \"payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl\" (UID: \"75a705b7-eccf-42e1-b238-e98a5018d2ca\") " pod="openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl" Sep 10 16:19:04.605900 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.605872 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a705b7-eccf-42e1-b238-e98a5018d2ca-plugins-config-volume\") pod \"payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl\" (UID: \"75a705b7-eccf-42e1-b238-e98a5018d2ca\") " pod="openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl" Sep 10 16:19:04.615178 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.615154 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwbgl\" (UniqueName: \"kubernetes.io/projected/75a705b7-eccf-42e1-b238-e98a5018d2ca-kube-api-access-wwbgl\") pod \"payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl\" (UID: \"75a705b7-eccf-42e1-b238-e98a5018d2ca\") " pod="openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl" Sep 10 16:19:04.706615 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.706562 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bdfa708-7fd8-44bd-a474-8726fbe75466-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4\" (UID: \"3bdfa708-7fd8-44bd-a474-8726fbe75466\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4" Sep 10 16:19:04.706803 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.706675 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8qhl\" (UniqueName: \"kubernetes.io/projected/3bdfa708-7fd8-44bd-a474-8726fbe75466-kube-api-access-b8qhl\") pod \"payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4\" (UID: \"3bdfa708-7fd8-44bd-a474-8726fbe75466\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4" Sep 10 16:19:04.707272 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.707252 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bdfa708-7fd8-44bd-a474-8726fbe75466-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4\" (UID: \"3bdfa708-7fd8-44bd-a474-8726fbe75466\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4" Sep 10 16:19:04.713770 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.713745 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl" Sep 10 16:19:04.716396 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.716375 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8qhl\" (UniqueName: \"kubernetes.io/projected/3bdfa708-7fd8-44bd-a474-8726fbe75466-kube-api-access-b8qhl\") pod \"payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4\" (UID: \"3bdfa708-7fd8-44bd-a474-8726fbe75466\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4" Sep 10 16:19:04.766431 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.766408 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4" Sep 10 16:19:04.904143 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.904099 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:04.989839 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.989807 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl"] Sep 10 16:19:04.991752 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:19:04.991725 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a705b7_eccf_42e1_b238_e98a5018d2ca.slice/crio-bb15877c6d0368b35757cf4c1570c5836f6fe8170552e63e1fd1fd9e4661f963 WatchSource:0}: Error finding container bb15877c6d0368b35757cf4c1570c5836f6fe8170552e63e1fd1fd9e4661f963: Status 404 returned error can't find the container with id bb15877c6d0368b35757cf4c1570c5836f6fe8170552e63e1fd1fd9e4661f963 Sep 10 16:19:04.993240 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:04.993208 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:05.051507 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:05.051479 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4"] Sep 10 16:19:05.053992 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:19:05.053968 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bdfa708_7fd8_44bd_a474_8726fbe75466.slice/crio-f0b8600113d16373d9016ff294fc9aa7d4f557622caf366dca2c652c07d68cfc WatchSource:0}: Error finding container f0b8600113d16373d9016ff294fc9aa7d4f557622caf366dca2c652c07d68cfc: Status 404 returned error can't find the container with id f0b8600113d16373d9016ff294fc9aa7d4f557622caf366dca2c652c07d68cfc Sep 10 16:19:05.065389 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:05.065351 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4" event={"ID":"3bdfa708-7fd8-44bd-a474-8726fbe75466","Type":"ContainerStarted","Data":"f0b8600113d16373d9016ff294fc9aa7d4f557622caf366dca2c652c07d68cfc"} Sep 10 16:19:05.067276 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:05.067241 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl" event={"ID":"75a705b7-eccf-42e1-b238-e98a5018d2ca","Type":"ContainerStarted","Data":"bb15877c6d0368b35757cf4c1570c5836f6fe8170552e63e1fd1fd9e4661f963"} Sep 10 16:19:05.168446 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:05.168360 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:05.522463 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:05.522374 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:05.625118 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:05.625031 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:05.682605 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:05.682549 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc"] Sep 10 16:19:05.686763 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:05.686741 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc" Sep 10 16:19:05.689124 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:05.689096 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-fefaa7ab-dockercfg-qh5vp\"" Sep 10 16:19:05.689607 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:05.689584 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-fefaa7ab\"" Sep 10 16:19:05.699680 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:05.699650 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc"] Sep 10 16:19:05.816448 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:05.816350 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c75s9\" (UniqueName: \"kubernetes.io/projected/0e29b74a-80c7-45d7-b7ff-916529b38b41-kube-api-access-c75s9\") pod \"payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc\" (UID: \"0e29b74a-80c7-45d7-b7ff-916529b38b41\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc" Sep 10 16:19:05.816626 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:05.816525 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e29b74a-80c7-45d7-b7ff-916529b38b41-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc\" (UID: \"0e29b74a-80c7-45d7-b7ff-916529b38b41\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc" Sep 10 16:19:05.917889 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:05.917854 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e29b74a-80c7-45d7-b7ff-916529b38b41-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc\" (UID: \"0e29b74a-80c7-45d7-b7ff-916529b38b41\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc" Sep 10 16:19:05.918074 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:05.917930 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c75s9\" (UniqueName: \"kubernetes.io/projected/0e29b74a-80c7-45d7-b7ff-916529b38b41-kube-api-access-c75s9\") pod \"payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc\" (UID: \"0e29b74a-80c7-45d7-b7ff-916529b38b41\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc" Sep 10 16:19:05.918705 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:05.918683 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e29b74a-80c7-45d7-b7ff-916529b38b41-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc\" (UID: \"0e29b74a-80c7-45d7-b7ff-916529b38b41\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc" Sep 10 16:19:05.931525 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:05.931495 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c75s9\" (UniqueName: \"kubernetes.io/projected/0e29b74a-80c7-45d7-b7ff-916529b38b41-kube-api-access-c75s9\") pod \"payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc\" (UID: \"0e29b74a-80c7-45d7-b7ff-916529b38b41\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc" Sep 10 16:19:05.966484 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:05.966427 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:06.004724 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:06.004685 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc" Sep 10 16:19:06.079527 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:06.079449 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4" event={"ID":"3bdfa708-7fd8-44bd-a474-8726fbe75466","Type":"ContainerStarted","Data":"4fcfe316f15f0dd6b2672c53f298fb991cab249ed926554ad1568705215f2292"} Sep 10 16:19:06.080366 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:06.080242 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4" Sep 10 16:19:06.084167 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:06.083475 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl" event={"ID":"75a705b7-eccf-42e1-b238-e98a5018d2ca","Type":"ContainerStarted","Data":"b6b056f22f614363990b39393ca6d08c7113eaf4155eebb25f3bc2474c48226c"} Sep 10 16:19:06.084167 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:06.083984 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl" Sep 10 16:19:06.100578 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:06.100509 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4" podStartSLOduration=2.100488751 podStartE2EDuration="2.100488751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:19:06.099432079 +0000 UTC m=+1045.622677277" watchObservedRunningTime="2026-09-10 16:19:06.100488751 +0000 UTC m=+1045.623733998" Sep 10 16:19:06.117379 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:06.117326 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl" podStartSLOduration=2.117273373 podStartE2EDuration="2.117273373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:19:06.116342597 +0000 UTC m=+1045.639587797" watchObservedRunningTime="2026-09-10 16:19:06.117273373 +0000 UTC m=+1045.640518572" Sep 10 16:19:06.297065 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:06.297025 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc"] Sep 10 16:19:06.302369 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:19:06.302290 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e29b74a_80c7_45d7_b7ff_916529b38b41.slice/crio-ec1a33a3e7d784a411a1223f817fc5a35f7ff297a352802288af142dddc3380b WatchSource:0}: Error finding container ec1a33a3e7d784a411a1223f817fc5a35f7ff297a352802288af142dddc3380b: Status 404 returned error can't find the container with id ec1a33a3e7d784a411a1223f817fc5a35f7ff297a352802288af142dddc3380b Sep 10 16:19:07.012715 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:07.012668 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:07.092258 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:07.092222 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc" event={"ID":"0e29b74a-80c7-45d7-b7ff-916529b38b41","Type":"ContainerStarted","Data":"023c2e3710ae4eca5d632ce2b7e1d77ae5d027658282fc2ae39c92487f183ef7"} Sep 10 16:19:07.092480 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:07.092285 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc" event={"ID":"0e29b74a-80c7-45d7-b7ff-916529b38b41","Type":"ContainerStarted","Data":"ec1a33a3e7d784a411a1223f817fc5a35f7ff297a352802288af142dddc3380b"} Sep 10 16:19:07.092872 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:07.092847 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc" Sep 10 16:19:07.121903 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:07.121768 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc" podStartSLOduration=2.121747224 podStartE2EDuration="2.121747224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:19:07.120814048 +0000 UTC m=+1046.644059247" watchObservedRunningTime="2026-09-10 16:19:07.121747224 +0000 UTC m=+1046.644992422" Sep 10 16:19:08.025692 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:08.025661 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5" Sep 10 16:19:08.826344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:08.826292 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:09.175978 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:09.175943 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:19:11.253579 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:11.253541 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:11.365530 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:11.365499 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:11.608775 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:11.608690 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:11.748103 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:11.748074 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:13.831088 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:13.831052 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:13.923176 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:13.923136 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:14.090203 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:14.090126 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:14.324272 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:14.324224 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:16.851229 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:16.851185 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:16.954127 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:16.954100 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:18.101954 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:18.101924 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl" Sep 10 16:19:18.102455 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:18.102438 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4" Sep 10 16:19:18.375235 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:18.375200 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:18.716343 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:18.716212 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:19.047629 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:19.047534 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:19.110230 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:19.110202 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc" Sep 10 16:19:19.604713 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:19.604675 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:21.968138 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:21.968100 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:22.081665 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:22.081630 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:22.235152 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:22.235058 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:22.335657 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:22.335625 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:22.796144 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:22.796110 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:22.891179 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:22.891147 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:24.040267 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.040228 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:24.140841 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.140808 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:24.576237 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.576204 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:24.674032 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.673999 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:24.694527 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.694496 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn"] Sep 10 16:19:24.698617 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.698595 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn" Sep 10 16:19:24.701316 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.701275 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e22c72\"" Sep 10 16:19:24.702663 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.702640 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m"] Sep 10 16:19:24.714459 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.713535 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn"] Sep 10 16:19:24.715403 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.714253 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m" Sep 10 16:19:24.719883 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.719804 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m"] Sep 10 16:19:24.719883 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.719817 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e22c72-dockercfg-2cdng\"" Sep 10 16:19:24.779623 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.779588 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd94c4ff-742e-4143-b72f-cbbade05817e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn\" (UID: \"dd94c4ff-742e-4143-b72f-cbbade05817e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn" Sep 10 16:19:24.779781 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.779633 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz589\" (UniqueName: \"kubernetes.io/projected/dd94c4ff-742e-4143-b72f-cbbade05817e-kube-api-access-jz589\") pod \"payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn\" (UID: \"dd94c4ff-742e-4143-b72f-cbbade05817e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn" Sep 10 16:19:24.880426 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.880386 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7drfk\" (UniqueName: \"kubernetes.io/projected/8aae576e-3c67-4e21-95af-9895a2a305cf-kube-api-access-7drfk\") pod \"payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m\" (UID: \"8aae576e-3c67-4e21-95af-9895a2a305cf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m" Sep 10 16:19:24.880602 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.880484 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd94c4ff-742e-4143-b72f-cbbade05817e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn\" (UID: \"dd94c4ff-742e-4143-b72f-cbbade05817e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn" Sep 10 16:19:24.880602 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.880506 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz589\" (UniqueName: \"kubernetes.io/projected/dd94c4ff-742e-4143-b72f-cbbade05817e-kube-api-access-jz589\") pod \"payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn\" (UID: \"dd94c4ff-742e-4143-b72f-cbbade05817e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn" Sep 10 16:19:24.880723 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.880674 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aae576e-3c67-4e21-95af-9895a2a305cf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m\" (UID: \"8aae576e-3c67-4e21-95af-9895a2a305cf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m" Sep 10 16:19:24.881366 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.881343 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd94c4ff-742e-4143-b72f-cbbade05817e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn\" (UID: \"dd94c4ff-742e-4143-b72f-cbbade05817e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn" Sep 10 16:19:24.889746 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.889719 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz589\" (UniqueName: \"kubernetes.io/projected/dd94c4ff-742e-4143-b72f-cbbade05817e-kube-api-access-jz589\") pod \"payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn\" (UID: \"dd94c4ff-742e-4143-b72f-cbbade05817e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn" Sep 10 16:19:24.981356 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.981323 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aae576e-3c67-4e21-95af-9895a2a305cf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m\" (UID: \"8aae576e-3c67-4e21-95af-9895a2a305cf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m" Sep 10 16:19:24.981544 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.981424 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7drfk\" (UniqueName: \"kubernetes.io/projected/8aae576e-3c67-4e21-95af-9895a2a305cf-kube-api-access-7drfk\") pod \"payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m\" (UID: \"8aae576e-3c67-4e21-95af-9895a2a305cf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m" Sep 10 16:19:24.982155 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.982134 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aae576e-3c67-4e21-95af-9895a2a305cf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m\" (UID: \"8aae576e-3c67-4e21-95af-9895a2a305cf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m" Sep 10 16:19:24.999008 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:24.998982 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7drfk\" (UniqueName: \"kubernetes.io/projected/8aae576e-3c67-4e21-95af-9895a2a305cf-kube-api-access-7drfk\") pod \"payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m\" (UID: \"8aae576e-3c67-4e21-95af-9895a2a305cf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m" Sep 10 16:19:25.018523 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:25.018497 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn" Sep 10 16:19:25.029973 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:25.029953 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m" Sep 10 16:19:25.205566 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:25.205477 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:25.298845 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:19:25.298816 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd94c4ff_742e_4143_b72f_cbbade05817e.slice/crio-a3f2600efa4e9a34584193a106d5bce892a519c02b92fab92cac586792860f46 WatchSource:0}: Error finding container a3f2600efa4e9a34584193a106d5bce892a519c02b92fab92cac586792860f46: Status 404 returned error can't find the container with id a3f2600efa4e9a34584193a106d5bce892a519c02b92fab92cac586792860f46 Sep 10 16:19:25.299265 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:25.299241 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn"] Sep 10 16:19:25.324778 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:25.324752 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m"] Sep 10 16:19:25.333416 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:19:25.333394 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aae576e_3c67_4e21_95af_9895a2a305cf.slice/crio-9b76778cdd611fe3c12c89183375abc881a7e4698f953937e16ebce57aa2bf00 WatchSource:0}: Error finding container 9b76778cdd611fe3c12c89183375abc881a7e4698f953937e16ebce57aa2bf00: Status 404 returned error can't find the container with id 9b76778cdd611fe3c12c89183375abc881a7e4698f953937e16ebce57aa2bf00 Sep 10 16:19:25.420482 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:25.420431 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:26.210112 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:26.210069 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m" event={"ID":"8aae576e-3c67-4e21-95af-9895a2a305cf","Type":"ContainerStarted","Data":"9ca234dcb0c2169ce3bb01db5f0d7358a58551dde0d3cee8f7ed372fa98dc36c"} Sep 10 16:19:26.210607 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:26.210129 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m" event={"ID":"8aae576e-3c67-4e21-95af-9895a2a305cf","Type":"ContainerStarted","Data":"9b76778cdd611fe3c12c89183375abc881a7e4698f953937e16ebce57aa2bf00"} Sep 10 16:19:26.210607 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:26.210184 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m" Sep 10 16:19:26.212017 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:26.211990 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn" event={"ID":"dd94c4ff-742e-4143-b72f-cbbade05817e","Type":"ContainerStarted","Data":"378c835f1d11f27b8eda363b10d067b54608d54ae53388a8b29b0f6082f539f7"} Sep 10 16:19:26.212139 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:26.212038 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn" event={"ID":"dd94c4ff-742e-4143-b72f-cbbade05817e","Type":"ContainerStarted","Data":"a3f2600efa4e9a34584193a106d5bce892a519c02b92fab92cac586792860f46"} Sep 10 16:19:26.212139 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:26.212092 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn" Sep 10 16:19:26.220662 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:26.220639 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:26.240604 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:26.240550 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m" podStartSLOduration=2.240535827 podStartE2EDuration="2.240535827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:19:26.238439842 +0000 UTC m=+1065.761685041" watchObservedRunningTime="2026-09-10 16:19:26.240535827 +0000 UTC m=+1065.763781016" Sep 10 16:19:26.263292 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:26.263253 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn" podStartSLOduration=2.263240023 podStartE2EDuration="2.263240023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:19:26.260741334 +0000 UTC m=+1065.783986532" watchObservedRunningTime="2026-09-10 16:19:26.263240023 +0000 UTC m=+1065.786485220" Sep 10 16:19:27.090435 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:27.090398 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:27.287580 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:27.287531 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:27.450576 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:27.450542 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:27.659084 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:27.659052 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:19:28.113044 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:28.112986 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:29.057275 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:29.057240 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:30.275654 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:30.275623 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:30.366568 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:30.366535 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:37.220802 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:37.220714 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m" Sep 10 16:19:37.221252 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:37.220889 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn" Sep 10 16:19:45.156512 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:45.156480 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:45.874174 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:45.874138 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:45.918060 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:45.918024 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc"] Sep 10 16:19:45.918327 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:45.918276 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc" podUID="0e29b74a-80c7-45d7-b7ff-916529b38b41" containerName="payload-pre-processing" containerID="cri-o://023c2e3710ae4eca5d632ce2b7e1d77ae5d027658282fc2ae39c92487f183ef7" gracePeriod=30 Sep 10 16:19:45.968990 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:45.968960 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:46.069710 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:46.069670 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:46.171280 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:46.171235 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:46.173672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:46.173655 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc" Sep 10 16:19:46.274975 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:46.274937 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e29b74a-80c7-45d7-b7ff-916529b38b41-plugins-config-volume\") pod \"0e29b74a-80c7-45d7-b7ff-916529b38b41\" (UID: \"0e29b74a-80c7-45d7-b7ff-916529b38b41\") " Sep 10 16:19:46.275155 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:46.275063 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c75s9\" (UniqueName: \"kubernetes.io/projected/0e29b74a-80c7-45d7-b7ff-916529b38b41-kube-api-access-c75s9\") pod \"0e29b74a-80c7-45d7-b7ff-916529b38b41\" (UID: \"0e29b74a-80c7-45d7-b7ff-916529b38b41\") " Sep 10 16:19:46.275441 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:46.275415 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e29b74a-80c7-45d7-b7ff-916529b38b41-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0e29b74a-80c7-45d7-b7ff-916529b38b41" (UID: "0e29b74a-80c7-45d7-b7ff-916529b38b41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:19:46.277413 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:46.277386 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e29b74a-80c7-45d7-b7ff-916529b38b41-kube-api-access-c75s9" (OuterVolumeSpecName: "kube-api-access-c75s9") pod "0e29b74a-80c7-45d7-b7ff-916529b38b41" (UID: "0e29b74a-80c7-45d7-b7ff-916529b38b41"). InnerVolumeSpecName "kube-api-access-c75s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:19:46.291164 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:46.291138 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:46.341718 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:46.341680 2541 generic.go:356] "Generic (PLEG): container finished" podID="0e29b74a-80c7-45d7-b7ff-916529b38b41" containerID="023c2e3710ae4eca5d632ce2b7e1d77ae5d027658282fc2ae39c92487f183ef7" exitCode=0 Sep 10 16:19:46.341920 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:46.341736 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc" Sep 10 16:19:46.341920 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:46.341760 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc" event={"ID":"0e29b74a-80c7-45d7-b7ff-916529b38b41","Type":"ContainerDied","Data":"023c2e3710ae4eca5d632ce2b7e1d77ae5d027658282fc2ae39c92487f183ef7"} Sep 10 16:19:46.341920 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:46.341811 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc" event={"ID":"0e29b74a-80c7-45d7-b7ff-916529b38b41","Type":"ContainerDied","Data":"ec1a33a3e7d784a411a1223f817fc5a35f7ff297a352802288af142dddc3380b"} Sep 10 16:19:46.341920 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:46.341844 2541 scope.go:117] "RemoveContainer" containerID="023c2e3710ae4eca5d632ce2b7e1d77ae5d027658282fc2ae39c92487f183ef7" Sep 10 16:19:46.351783 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:46.351768 2541 scope.go:117] "RemoveContainer" containerID="023c2e3710ae4eca5d632ce2b7e1d77ae5d027658282fc2ae39c92487f183ef7" Sep 10 16:19:46.352048 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:19:46.352027 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"023c2e3710ae4eca5d632ce2b7e1d77ae5d027658282fc2ae39c92487f183ef7\": container with ID starting with 023c2e3710ae4eca5d632ce2b7e1d77ae5d027658282fc2ae39c92487f183ef7 not found: ID does not exist" containerID="023c2e3710ae4eca5d632ce2b7e1d77ae5d027658282fc2ae39c92487f183ef7" Sep 10 16:19:46.352097 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:46.352058 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"023c2e3710ae4eca5d632ce2b7e1d77ae5d027658282fc2ae39c92487f183ef7"} err="failed to get container status \"023c2e3710ae4eca5d632ce2b7e1d77ae5d027658282fc2ae39c92487f183ef7\": rpc error: code = NotFound desc = could not find container \"023c2e3710ae4eca5d632ce2b7e1d77ae5d027658282fc2ae39c92487f183ef7\": container with ID starting with 023c2e3710ae4eca5d632ce2b7e1d77ae5d027658282fc2ae39c92487f183ef7 not found: ID does not exist" Sep 10 16:19:46.366423 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:46.366398 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc"] Sep 10 16:19:46.368703 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:46.368679 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fefaa7ab-58f8d84c8-k7zrc"] Sep 10 16:19:46.375692 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:46.375674 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c75s9\" (UniqueName: \"kubernetes.io/projected/0e29b74a-80c7-45d7-b7ff-916529b38b41-kube-api-access-c75s9\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:19:46.375751 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:46.375695 2541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e29b74a-80c7-45d7-b7ff-916529b38b41-plugins-config-volume\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:19:46.444175 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:46.444077 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:46.919605 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:46.919575 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:46.980246 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:46.980212 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e29b74a-80c7-45d7-b7ff-916529b38b41" path="/var/lib/kubelet/pods/0e29b74a-80c7-45d7-b7ff-916529b38b41/volumes" Sep 10 16:19:47.064728 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:47.064663 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:47.250331 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:47.250232 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:47.533895 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:47.533781 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:47.688078 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:47.688031 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:47.842068 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:47.841987 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:50.408868 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:50.408834 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:50.511324 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:50.511260 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:50.610744 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:50.610714 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:50.710234 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:50.710156 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:51.054183 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:51.054109 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:51.227601 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:51.227554 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:51.322589 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:51.322503 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:51.355389 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:51.355349 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj"] Sep 10 16:19:51.355802 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:51.355787 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e29b74a-80c7-45d7-b7ff-916529b38b41" containerName="payload-pre-processing" Sep 10 16:19:51.355848 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:51.355805 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e29b74a-80c7-45d7-b7ff-916529b38b41" containerName="payload-pre-processing" Sep 10 16:19:51.355896 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:51.355887 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e29b74a-80c7-45d7-b7ff-916529b38b41" containerName="payload-pre-processing" Sep 10 16:19:51.359984 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:51.359968 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj" Sep 10 16:19:51.362352 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:51.362333 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b7764c7d\"" Sep 10 16:19:51.362480 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:51.362387 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b7764c7d-dockercfg-b7wzm\"" Sep 10 16:19:51.372807 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:51.372777 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj"] Sep 10 16:19:51.430101 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:51.430069 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279c8e15-63f3-4c38-ad59-4702e2a669f4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj\" (UID: \"279c8e15-63f3-4c38-ad59-4702e2a669f4\") " pod="openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj" Sep 10 16:19:51.430507 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:51.430108 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwwfh\" (UniqueName: \"kubernetes.io/projected/279c8e15-63f3-4c38-ad59-4702e2a669f4-kube-api-access-cwwfh\") pod \"payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj\" (UID: \"279c8e15-63f3-4c38-ad59-4702e2a669f4\") " pod="openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj" Sep 10 16:19:51.531315 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:51.531257 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279c8e15-63f3-4c38-ad59-4702e2a669f4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj\" (UID: \"279c8e15-63f3-4c38-ad59-4702e2a669f4\") " pod="openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj" Sep 10 16:19:51.531315 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:51.531310 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwwfh\" (UniqueName: \"kubernetes.io/projected/279c8e15-63f3-4c38-ad59-4702e2a669f4-kube-api-access-cwwfh\") pod \"payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj\" (UID: \"279c8e15-63f3-4c38-ad59-4702e2a669f4\") " pod="openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj" Sep 10 16:19:51.531977 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:51.531956 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279c8e15-63f3-4c38-ad59-4702e2a669f4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj\" (UID: \"279c8e15-63f3-4c38-ad59-4702e2a669f4\") " pod="openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj" Sep 10 16:19:51.544543 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:51.544521 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwwfh\" (UniqueName: \"kubernetes.io/projected/279c8e15-63f3-4c38-ad59-4702e2a669f4-kube-api-access-cwwfh\") pod \"payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj\" (UID: \"279c8e15-63f3-4c38-ad59-4702e2a669f4\") " pod="openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj" Sep 10 16:19:51.672429 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:51.672390 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj" Sep 10 16:19:51.950290 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:51.950264 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj"] Sep 10 16:19:51.951922 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:19:51.951895 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279c8e15_63f3_4c38_ad59_4702e2a669f4.slice/crio-4e043d98b63b86d7ae24c68e7fddb5fadfaac3998331b64721936943645265de WatchSource:0}: Error finding container 4e043d98b63b86d7ae24c68e7fddb5fadfaac3998331b64721936943645265de: Status 404 returned error can't find the container with id 4e043d98b63b86d7ae24c68e7fddb5fadfaac3998331b64721936943645265de Sep 10 16:19:52.380137 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:52.380099 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj" event={"ID":"279c8e15-63f3-4c38-ad59-4702e2a669f4","Type":"ContainerStarted","Data":"35283f137837b182b256304f023ebe70cade8a88cfe6ae2e92380b5b433e4b58"} Sep 10 16:19:52.380364 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:52.380155 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj" event={"ID":"279c8e15-63f3-4c38-ad59-4702e2a669f4","Type":"ContainerStarted","Data":"4e043d98b63b86d7ae24c68e7fddb5fadfaac3998331b64721936943645265de"} Sep 10 16:19:52.380364 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:52.380177 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj" Sep 10 16:19:52.404962 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:52.404906 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj" podStartSLOduration=1.404888132 podStartE2EDuration="1.404888132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:19:52.402378848 +0000 UTC m=+1091.925624046" watchObservedRunningTime="2026-09-10 16:19:52.404888132 +0000 UTC m=+1091.928133343" Sep 10 16:19:55.145155 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:19:55.145113 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:20:00.351238 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:00.351204 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:03.388665 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:03.388637 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj" Sep 10 16:20:05.299572 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:05.299545 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:12.554705 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:12.554674 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:20:19.006835 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:19.006799 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:21.635814 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:21.635756 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:21.643871 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:21.643845 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj"] Sep 10 16:20:21.644338 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:21.644267 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj" podUID="279c8e15-63f3-4c38-ad59-4702e2a669f4" containerName="payload-processing" containerID="cri-o://35283f137837b182b256304f023ebe70cade8a88cfe6ae2e92380b5b433e4b58" gracePeriod=30 Sep 10 16:20:21.738004 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:21.737968 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:21.838796 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:21.838763 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:21.889943 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:21.889881 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj" Sep 10 16:20:21.928495 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:21.928406 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:22.017496 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:22.017467 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwwfh\" (UniqueName: \"kubernetes.io/projected/279c8e15-63f3-4c38-ad59-4702e2a669f4-kube-api-access-cwwfh\") pod \"279c8e15-63f3-4c38-ad59-4702e2a669f4\" (UID: \"279c8e15-63f3-4c38-ad59-4702e2a669f4\") " Sep 10 16:20:22.017675 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:22.017522 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279c8e15-63f3-4c38-ad59-4702e2a669f4-plugins-config-volume\") pod \"279c8e15-63f3-4c38-ad59-4702e2a669f4\" (UID: \"279c8e15-63f3-4c38-ad59-4702e2a669f4\") " Sep 10 16:20:22.017976 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:22.017947 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/279c8e15-63f3-4c38-ad59-4702e2a669f4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "279c8e15-63f3-4c38-ad59-4702e2a669f4" (UID: "279c8e15-63f3-4c38-ad59-4702e2a669f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:20:22.019668 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:22.019641 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279c8e15-63f3-4c38-ad59-4702e2a669f4-kube-api-access-cwwfh" (OuterVolumeSpecName: "kube-api-access-cwwfh") pod "279c8e15-63f3-4c38-ad59-4702e2a669f4" (UID: "279c8e15-63f3-4c38-ad59-4702e2a669f4"). InnerVolumeSpecName "kube-api-access-cwwfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:20:22.021444 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:22.021419 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:22.118411 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:22.118383 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwwfh\" (UniqueName: \"kubernetes.io/projected/279c8e15-63f3-4c38-ad59-4702e2a669f4-kube-api-access-cwwfh\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:20:22.118612 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:22.118598 2541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279c8e15-63f3-4c38-ad59-4702e2a669f4-plugins-config-volume\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:20:22.467023 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:22.466996 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:22.557032 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:22.556994 2541 generic.go:356] "Generic (PLEG): container finished" podID="279c8e15-63f3-4c38-ad59-4702e2a669f4" containerID="35283f137837b182b256304f023ebe70cade8a88cfe6ae2e92380b5b433e4b58" exitCode=0 Sep 10 16:20:22.557176 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:22.557037 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj" Sep 10 16:20:22.557176 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:22.557084 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj" event={"ID":"279c8e15-63f3-4c38-ad59-4702e2a669f4","Type":"ContainerDied","Data":"35283f137837b182b256304f023ebe70cade8a88cfe6ae2e92380b5b433e4b58"} Sep 10 16:20:22.557176 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:22.557128 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj" event={"ID":"279c8e15-63f3-4c38-ad59-4702e2a669f4","Type":"ContainerDied","Data":"4e043d98b63b86d7ae24c68e7fddb5fadfaac3998331b64721936943645265de"} Sep 10 16:20:22.557176 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:22.557151 2541 scope.go:117] "RemoveContainer" containerID="35283f137837b182b256304f023ebe70cade8a88cfe6ae2e92380b5b433e4b58" Sep 10 16:20:22.567283 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:22.567265 2541 scope.go:117] "RemoveContainer" containerID="35283f137837b182b256304f023ebe70cade8a88cfe6ae2e92380b5b433e4b58" Sep 10 16:20:22.567572 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:20:22.567550 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35283f137837b182b256304f023ebe70cade8a88cfe6ae2e92380b5b433e4b58\": container with ID starting with 35283f137837b182b256304f023ebe70cade8a88cfe6ae2e92380b5b433e4b58 not found: ID does not exist" containerID="35283f137837b182b256304f023ebe70cade8a88cfe6ae2e92380b5b433e4b58" Sep 10 16:20:22.567619 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:22.567583 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35283f137837b182b256304f023ebe70cade8a88cfe6ae2e92380b5b433e4b58"} err="failed to get container status \"35283f137837b182b256304f023ebe70cade8a88cfe6ae2e92380b5b433e4b58\": rpc error: code = NotFound desc = could not find container \"35283f137837b182b256304f023ebe70cade8a88cfe6ae2e92380b5b433e4b58\": container with ID starting with 35283f137837b182b256304f023ebe70cade8a88cfe6ae2e92380b5b433e4b58 not found: ID does not exist" Sep 10 16:20:22.582050 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:22.582016 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj"] Sep 10 16:20:22.588841 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:22.588816 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b7764c7d-765dbc6966-phfpj"] Sep 10 16:20:22.887828 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:22.887800 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:22.980879 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:22.980844 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="279c8e15-63f3-4c38-ad59-4702e2a669f4" path="/var/lib/kubelet/pods/279c8e15-63f3-4c38-ad59-4702e2a669f4/volumes" Sep 10 16:20:23.414038 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:23.414006 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:25.500148 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:25.500106 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:25.818552 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:25.818467 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:26.710049 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:26.710016 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:26.836982 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:26.836912 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:26.946828 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:26.946793 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:27.248175 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:27.248138 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:27.351250 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:27.351216 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:34.061174 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:34.061136 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:20:40.336059 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.336021 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9"] Sep 10 16:20:40.336503 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.336489 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="279c8e15-63f3-4c38-ad59-4702e2a669f4" containerName="payload-processing" Sep 10 16:20:40.336552 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.336507 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="279c8e15-63f3-4c38-ad59-4702e2a669f4" containerName="payload-processing" Sep 10 16:20:40.336587 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.336573 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="279c8e15-63f3-4c38-ad59-4702e2a669f4" containerName="payload-processing" Sep 10 16:20:40.339666 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.339651 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:40.348841 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.348823 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-37c805\"/\"kube-root-ca.crt\"" Sep 10 16:20:40.349429 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.349412 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-37c805\"/\"default-dockercfg-dwlv9\"" Sep 10 16:20:40.349485 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.349433 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-37c805\"/\"test-model-37c805-kserve-self-signed-certs\"" Sep 10 16:20:40.353724 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.353708 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-37c805\"/\"openshift-service-ca.crt\"" Sep 10 16:20:40.364614 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.364591 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9"] Sep 10 16:20:40.481535 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.481478 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:40.490508 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.490464 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-tmp-dir\") pod \"test-model-37c805-kserve-75465bbcd9-t9nm9\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:40.490508 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.490512 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-dshm\") pod \"test-model-37c805-kserve-75465bbcd9-t9nm9\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:40.490737 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.490564 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34fc915d-d815-4ec4-a521-051f2d697fce-tls-certs\") pod \"test-model-37c805-kserve-75465bbcd9-t9nm9\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:40.490737 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.490599 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-home\") pod \"test-model-37c805-kserve-75465bbcd9-t9nm9\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:40.490737 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.490644 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcs4j\" (UniqueName: \"kubernetes.io/projected/34fc915d-d815-4ec4-a521-051f2d697fce-kube-api-access-xcs4j\") pod \"test-model-37c805-kserve-75465bbcd9-t9nm9\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:40.490737 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.490681 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-model-cache\") pod \"test-model-37c805-kserve-75465bbcd9-t9nm9\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:40.591869 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.591782 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcs4j\" (UniqueName: \"kubernetes.io/projected/34fc915d-d815-4ec4-a521-051f2d697fce-kube-api-access-xcs4j\") pod \"test-model-37c805-kserve-75465bbcd9-t9nm9\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:40.591869 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.591822 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-model-cache\") pod \"test-model-37c805-kserve-75465bbcd9-t9nm9\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:40.592120 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.591883 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-tmp-dir\") pod \"test-model-37c805-kserve-75465bbcd9-t9nm9\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:40.592120 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.591901 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-dshm\") pod \"test-model-37c805-kserve-75465bbcd9-t9nm9\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:40.592120 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.591919 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34fc915d-d815-4ec4-a521-051f2d697fce-tls-certs\") pod \"test-model-37c805-kserve-75465bbcd9-t9nm9\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:40.592120 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.591935 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-home\") pod \"test-model-37c805-kserve-75465bbcd9-t9nm9\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:40.592449 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.592423 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-tmp-dir\") pod \"test-model-37c805-kserve-75465bbcd9-t9nm9\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:40.592622 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.592443 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-home\") pod \"test-model-37c805-kserve-75465bbcd9-t9nm9\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:40.592697 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.592556 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-model-cache\") pod \"test-model-37c805-kserve-75465bbcd9-t9nm9\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:40.594655 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.594621 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-dshm\") pod \"test-model-37c805-kserve-75465bbcd9-t9nm9\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:40.595007 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.594935 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34fc915d-d815-4ec4-a521-051f2d697fce-tls-certs\") pod \"test-model-37c805-kserve-75465bbcd9-t9nm9\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:40.607756 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.607734 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcs4j\" (UniqueName: \"kubernetes.io/projected/34fc915d-d815-4ec4-a521-051f2d697fce-kube-api-access-xcs4j\") pod \"test-model-37c805-kserve-75465bbcd9-t9nm9\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:40.649655 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.649608 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:40.931550 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:40.931526 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9"] Sep 10 16:20:40.934229 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:20:40.934204 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34fc915d_d815_4ec4_a521_051f2d697fce.slice/crio-8b3b3e72e4ce2f77e6d8097513b4e394147af0906aae852517dd730a396349d3 WatchSource:0}: Error finding container 8b3b3e72e4ce2f77e6d8097513b4e394147af0906aae852517dd730a396349d3: Status 404 returned error can't find the container with id 8b3b3e72e4ce2f77e6d8097513b4e394147af0906aae852517dd730a396349d3 Sep 10 16:20:41.673345 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:41.673283 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" event={"ID":"34fc915d-d815-4ec4-a521-051f2d697fce","Type":"ContainerStarted","Data":"fa0eb623f13cb15f56f779c1e0402135b6d4200fb71646eabdbd97ba6f4e46c6"} Sep 10 16:20:41.673752 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:41.673354 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" event={"ID":"34fc915d-d815-4ec4-a521-051f2d697fce","Type":"ContainerStarted","Data":"8b3b3e72e4ce2f77e6d8097513b4e394147af0906aae852517dd730a396349d3"} Sep 10 16:20:41.693150 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:41.693094 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" podStartSLOduration=1.693078056 podStartE2EDuration="1.693078056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:20:41.692421931 +0000 UTC m=+1141.215667128" watchObservedRunningTime="2026-09-10 16:20:41.693078056 +0000 UTC m=+1141.216323253" Sep 10 16:20:43.012474 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:43.012437 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:44.036671 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:44.036635 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:44.376147 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:44.376121 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:47.893289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:47.893248 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:50.649928 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:50.649889 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:50.650328 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:50.649950 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:50.662842 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:50.662820 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:50.738694 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:50.738664 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:20:51.315101 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:51.315063 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a64cdb4-gkwv7"] Sep 10 16:20:51.319992 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:51.319974 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a64cdb4-gkwv7" Sep 10 16:20:51.323864 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:51.323837 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l2dj2\"" Sep 10 16:20:51.324119 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:51.324101 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:20:51.334202 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:51.334173 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a64cdb4-gkwv7"] Sep 10 16:20:51.388124 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:51.388084 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e4d04b9-25b2-4e6b-b85a-2e574b2609f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a64cdb4-gkwv7\" (UID: \"0e4d04b9-25b2-4e6b-b85a-2e574b2609f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a64cdb4-gkwv7" Sep 10 16:20:51.488637 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:51.488585 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e4d04b9-25b2-4e6b-b85a-2e574b2609f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a64cdb4-gkwv7\" (UID: \"0e4d04b9-25b2-4e6b-b85a-2e574b2609f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a64cdb4-gkwv7" Sep 10 16:20:51.489243 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:51.489218 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e4d04b9-25b2-4e6b-b85a-2e574b2609f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a64cdb4-gkwv7\" (UID: \"0e4d04b9-25b2-4e6b-b85a-2e574b2609f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a64cdb4-gkwv7" Sep 10 16:20:51.631124 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:51.631086 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a64cdb4-gkwv7" Sep 10 16:20:51.916964 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:51.916937 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a64cdb4-gkwv7"] Sep 10 16:20:51.918734 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:20:51.918709 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4d04b9_25b2_4e6b_b85a_2e574b2609f5.slice/crio-718c30dc59aee22821b77b9cc1ba403f2c3d1d94c274bf6c7a567bcabadf8679 WatchSource:0}: Error finding container 718c30dc59aee22821b77b9cc1ba403f2c3d1d94c274bf6c7a567bcabadf8679: Status 404 returned error can't find the container with id 718c30dc59aee22821b77b9cc1ba403f2c3d1d94c274bf6c7a567bcabadf8679 Sep 10 16:20:52.160683 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:52.160651 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:20:52.739891 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:52.739861 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a64cdb4-gkwv7" event={"ID":"0e4d04b9-25b2-4e6b-b85a-2e574b2609f5","Type":"ContainerStarted","Data":"718c30dc59aee22821b77b9cc1ba403f2c3d1d94c274bf6c7a567bcabadf8679"} Sep 10 16:20:54.295885 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:54.295827 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:54.689492 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:54.689445 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:55.003693 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.003665 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:55.030792 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.030475 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl"] Sep 10 16:20:55.030792 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.030738 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl" podUID="75a705b7-eccf-42e1-b238-e98a5018d2ca" containerName="payload-processing" containerID="cri-o://b6b056f22f614363990b39393ca6d08c7113eaf4155eebb25f3bc2474c48226c" gracePeriod=30 Sep 10 16:20:55.036081 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.036047 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4"] Sep 10 16:20:55.036503 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.036469 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4" podUID="3bdfa708-7fd8-44bd-a474-8726fbe75466" containerName="payload-pre-processing" containerID="cri-o://4fcfe316f15f0dd6b2672c53f298fb991cab249ed926554ad1568705215f2292" gracePeriod=30 Sep 10 16:20:55.102823 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.102789 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:55.214838 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.214806 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:55.315180 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.315153 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:55.322529 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.322508 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl" Sep 10 16:20:55.348520 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.348497 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4" Sep 10 16:20:55.419600 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.419566 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwbgl\" (UniqueName: \"kubernetes.io/projected/75a705b7-eccf-42e1-b238-e98a5018d2ca-kube-api-access-wwbgl\") pod \"75a705b7-eccf-42e1-b238-e98a5018d2ca\" (UID: \"75a705b7-eccf-42e1-b238-e98a5018d2ca\") " Sep 10 16:20:55.419749 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.419727 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8qhl\" (UniqueName: \"kubernetes.io/projected/3bdfa708-7fd8-44bd-a474-8726fbe75466-kube-api-access-b8qhl\") pod \"3bdfa708-7fd8-44bd-a474-8726fbe75466\" (UID: \"3bdfa708-7fd8-44bd-a474-8726fbe75466\") " Sep 10 16:20:55.419806 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.419768 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bdfa708-7fd8-44bd-a474-8726fbe75466-plugins-config-volume\") pod \"3bdfa708-7fd8-44bd-a474-8726fbe75466\" (UID: \"3bdfa708-7fd8-44bd-a474-8726fbe75466\") " Sep 10 16:20:55.419862 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.419815 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a705b7-eccf-42e1-b238-e98a5018d2ca-plugins-config-volume\") pod \"75a705b7-eccf-42e1-b238-e98a5018d2ca\" (UID: \"75a705b7-eccf-42e1-b238-e98a5018d2ca\") " Sep 10 16:20:55.420181 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.420152 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bdfa708-7fd8-44bd-a474-8726fbe75466-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3bdfa708-7fd8-44bd-a474-8726fbe75466" (UID: "3bdfa708-7fd8-44bd-a474-8726fbe75466"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:20:55.420315 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.420269 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a705b7-eccf-42e1-b238-e98a5018d2ca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "75a705b7-eccf-42e1-b238-e98a5018d2ca" (UID: "75a705b7-eccf-42e1-b238-e98a5018d2ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:20:55.422064 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.422044 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bdfa708-7fd8-44bd-a474-8726fbe75466-kube-api-access-b8qhl" (OuterVolumeSpecName: "kube-api-access-b8qhl") pod "3bdfa708-7fd8-44bd-a474-8726fbe75466" (UID: "3bdfa708-7fd8-44bd-a474-8726fbe75466"). InnerVolumeSpecName "kube-api-access-b8qhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:20:55.422172 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.422143 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a705b7-eccf-42e1-b238-e98a5018d2ca-kube-api-access-wwbgl" (OuterVolumeSpecName: "kube-api-access-wwbgl") pod "75a705b7-eccf-42e1-b238-e98a5018d2ca" (UID: "75a705b7-eccf-42e1-b238-e98a5018d2ca"). InnerVolumeSpecName "kube-api-access-wwbgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:20:55.520895 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.520799 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8qhl\" (UniqueName: \"kubernetes.io/projected/3bdfa708-7fd8-44bd-a474-8726fbe75466-kube-api-access-b8qhl\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:20:55.520895 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.520833 2541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bdfa708-7fd8-44bd-a474-8726fbe75466-plugins-config-volume\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:20:55.520895 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.520845 2541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a705b7-eccf-42e1-b238-e98a5018d2ca-plugins-config-volume\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:20:55.520895 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.520856 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwbgl\" (UniqueName: \"kubernetes.io/projected/75a705b7-eccf-42e1-b238-e98a5018d2ca-kube-api-access-wwbgl\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:20:55.650165 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.650135 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:55.760893 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.760863 2541 generic.go:356] "Generic (PLEG): container finished" podID="3bdfa708-7fd8-44bd-a474-8726fbe75466" containerID="4fcfe316f15f0dd6b2672c53f298fb991cab249ed926554ad1568705215f2292" exitCode=0 Sep 10 16:20:55.761099 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.760919 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4" Sep 10 16:20:55.761099 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.760949 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4" event={"ID":"3bdfa708-7fd8-44bd-a474-8726fbe75466","Type":"ContainerDied","Data":"4fcfe316f15f0dd6b2672c53f298fb991cab249ed926554ad1568705215f2292"} Sep 10 16:20:55.761099 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.761004 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4" event={"ID":"3bdfa708-7fd8-44bd-a474-8726fbe75466","Type":"ContainerDied","Data":"f0b8600113d16373d9016ff294fc9aa7d4f557622caf366dca2c652c07d68cfc"} Sep 10 16:20:55.761099 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.761031 2541 scope.go:117] "RemoveContainer" containerID="4fcfe316f15f0dd6b2672c53f298fb991cab249ed926554ad1568705215f2292" Sep 10 16:20:55.762370 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.762348 2541 generic.go:356] "Generic (PLEG): container finished" podID="75a705b7-eccf-42e1-b238-e98a5018d2ca" containerID="b6b056f22f614363990b39393ca6d08c7113eaf4155eebb25f3bc2474c48226c" exitCode=0 Sep 10 16:20:55.762479 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.762387 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl" event={"ID":"75a705b7-eccf-42e1-b238-e98a5018d2ca","Type":"ContainerDied","Data":"b6b056f22f614363990b39393ca6d08c7113eaf4155eebb25f3bc2474c48226c"} Sep 10 16:20:55.762479 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.762399 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl" Sep 10 16:20:55.762479 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.762440 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl" event={"ID":"75a705b7-eccf-42e1-b238-e98a5018d2ca","Type":"ContainerDied","Data":"bb15877c6d0368b35757cf4c1570c5836f6fe8170552e63e1fd1fd9e4661f963"} Sep 10 16:20:55.764398 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.764369 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a64cdb4-gkwv7" event={"ID":"0e4d04b9-25b2-4e6b-b85a-2e574b2609f5","Type":"ContainerStarted","Data":"0343f6f84e34d5c91938222bce992621c034165ba62e2701299a6bda0e350f91"} Sep 10 16:20:55.771250 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.771222 2541 scope.go:117] "RemoveContainer" containerID="4fcfe316f15f0dd6b2672c53f298fb991cab249ed926554ad1568705215f2292" Sep 10 16:20:55.771522 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:20:55.771492 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fcfe316f15f0dd6b2672c53f298fb991cab249ed926554ad1568705215f2292\": container with ID starting with 4fcfe316f15f0dd6b2672c53f298fb991cab249ed926554ad1568705215f2292 not found: ID does not exist" containerID="4fcfe316f15f0dd6b2672c53f298fb991cab249ed926554ad1568705215f2292" Sep 10 16:20:55.771620 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.771531 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fcfe316f15f0dd6b2672c53f298fb991cab249ed926554ad1568705215f2292"} err="failed to get container status \"4fcfe316f15f0dd6b2672c53f298fb991cab249ed926554ad1568705215f2292\": rpc error: code = NotFound desc = could not find container \"4fcfe316f15f0dd6b2672c53f298fb991cab249ed926554ad1568705215f2292\": container with ID starting with 4fcfe316f15f0dd6b2672c53f298fb991cab249ed926554ad1568705215f2292 not found: ID does not exist" Sep 10 16:20:55.771620 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.771552 2541 scope.go:117] "RemoveContainer" containerID="b6b056f22f614363990b39393ca6d08c7113eaf4155eebb25f3bc2474c48226c" Sep 10 16:20:55.779825 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.779809 2541 scope.go:117] "RemoveContainer" containerID="b6b056f22f614363990b39393ca6d08c7113eaf4155eebb25f3bc2474c48226c" Sep 10 16:20:55.780075 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:20:55.780057 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6b056f22f614363990b39393ca6d08c7113eaf4155eebb25f3bc2474c48226c\": container with ID starting with b6b056f22f614363990b39393ca6d08c7113eaf4155eebb25f3bc2474c48226c not found: ID does not exist" containerID="b6b056f22f614363990b39393ca6d08c7113eaf4155eebb25f3bc2474c48226c" Sep 10 16:20:55.780127 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.780082 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6b056f22f614363990b39393ca6d08c7113eaf4155eebb25f3bc2474c48226c"} err="failed to get container status \"b6b056f22f614363990b39393ca6d08c7113eaf4155eebb25f3bc2474c48226c\": rpc error: code = NotFound desc = could not find container \"b6b056f22f614363990b39393ca6d08c7113eaf4155eebb25f3bc2474c48226c\": container with ID starting with b6b056f22f614363990b39393ca6d08c7113eaf4155eebb25f3bc2474c48226c not found: ID does not exist" Sep 10 16:20:55.795718 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.788844 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a64cdb4-gkwv7" podStartSLOduration=1.7573225 podStartE2EDuration="4.788821865s" totalImagesPullingTime="3.031499365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:20:55.788823139 +0000 UTC m=+1155.312068337" watchObservedRunningTime="2026-09-10 16:20:55.788821865 +0000 UTC m=+1155.312067073" Sep 10 16:20:55.807503 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.807472 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4"] Sep 10 16:20:55.812012 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.811990 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1596ac-58ddd5f96b-thgn4"] Sep 10 16:20:55.823535 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.823508 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl"] Sep 10 16:20:55.826197 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:55.826177 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1596ac-5bcdb8f96c-lgkhl"] Sep 10 16:20:56.772376 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:56.772334 2541 generic.go:356] "Generic (PLEG): container finished" podID="0e4d04b9-25b2-4e6b-b85a-2e574b2609f5" containerID="0343f6f84e34d5c91938222bce992621c034165ba62e2701299a6bda0e350f91" exitCode=0 Sep 10 16:20:56.772839 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:56.772399 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a64cdb4-gkwv7" event={"ID":"0e4d04b9-25b2-4e6b-b85a-2e574b2609f5","Type":"ContainerDied","Data":"0343f6f84e34d5c91938222bce992621c034165ba62e2701299a6bda0e350f91"} Sep 10 16:20:56.979615 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:56.979583 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bdfa708-7fd8-44bd-a474-8726fbe75466" path="/var/lib/kubelet/pods/3bdfa708-7fd8-44bd-a474-8726fbe75466/volumes" Sep 10 16:20:56.979946 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:56.979934 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75a705b7-eccf-42e1-b238-e98a5018d2ca" path="/var/lib/kubelet/pods/75a705b7-eccf-42e1-b238-e98a5018d2ca/volumes" Sep 10 16:20:57.919362 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:57.919342 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a64cdb4-gkwv7" Sep 10 16:20:57.939693 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:57.939668 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e4d04b9-25b2-4e6b-b85a-2e574b2609f5-maas-api-service-ca\") pod \"0e4d04b9-25b2-4e6b-b85a-2e574b2609f5\" (UID: \"0e4d04b9-25b2-4e6b-b85a-2e574b2609f5\") " Sep 10 16:20:57.939964 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:57.939944 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e4d04b9-25b2-4e6b-b85a-2e574b2609f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e4d04b9-25b2-4e6b-b85a-2e574b2609f5" (UID: "0e4d04b9-25b2-4e6b-b85a-2e574b2609f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:20:58.041061 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:58.040985 2541 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e4d04b9-25b2-4e6b-b85a-2e574b2609f5-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:20:58.542569 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:58.542531 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:58.787003 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:58.786968 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a64cdb4-gkwv7" event={"ID":"0e4d04b9-25b2-4e6b-b85a-2e574b2609f5","Type":"ContainerDied","Data":"718c30dc59aee22821b77b9cc1ba403f2c3d1d94c274bf6c7a567bcabadf8679"} Sep 10 16:20:58.787003 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:58.787001 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a64cdb4-gkwv7" Sep 10 16:20:58.787208 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:20:58.787014 2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="718c30dc59aee22821b77b9cc1ba403f2c3d1d94c274bf6c7a567bcabadf8679" Sep 10 16:21:03.172024 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:03.171954 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:04.201548 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.201509 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6"] Sep 10 16:21:04.201927 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.201910 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bdfa708-7fd8-44bd-a474-8726fbe75466" containerName="payload-pre-processing" Sep 10 16:21:04.201927 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.201922 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bdfa708-7fd8-44bd-a474-8726fbe75466" containerName="payload-pre-processing" Sep 10 16:21:04.202007 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.201934 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e4d04b9-25b2-4e6b-b85a-2e574b2609f5" containerName="revoke-keys" Sep 10 16:21:04.202007 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.201940 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e4d04b9-25b2-4e6b-b85a-2e574b2609f5" containerName="revoke-keys" Sep 10 16:21:04.202007 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.201959 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75a705b7-eccf-42e1-b238-e98a5018d2ca" containerName="payload-processing" Sep 10 16:21:04.202007 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.201965 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="75a705b7-eccf-42e1-b238-e98a5018d2ca" containerName="payload-processing" Sep 10 16:21:04.202132 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.202027 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bdfa708-7fd8-44bd-a474-8726fbe75466" containerName="payload-pre-processing" Sep 10 16:21:04.202132 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.202035 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e4d04b9-25b2-4e6b-b85a-2e574b2609f5" containerName="revoke-keys" Sep 10 16:21:04.202132 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.202041 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="75a705b7-eccf-42e1-b238-e98a5018d2ca" containerName="payload-processing" Sep 10 16:21:04.205459 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.205438 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.209197 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.209178 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-555853-openshift-default-dockercfg-wj267\"" Sep 10 16:21:04.218150 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.218124 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6"] Sep 10 16:21:04.300568 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.300532 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-credential-socket\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.300741 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.300579 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-envoy\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.300741 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.300612 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-podinfo\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.300741 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.300658 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-workload-socket\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.300741 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.300684 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-token\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.300741 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.300702 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-data\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.300741 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.300718 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4a3115-4891-4117-82cd-7cbd804cef03-istiod-ca-cert\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.300741 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.300732 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvkpc\" (UniqueName: \"kubernetes.io/projected/fc4a3115-4891-4117-82cd-7cbd804cef03-kube-api-access-xvkpc\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.300994 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.300760 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-workload-certs\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.402248 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.402206 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-podinfo\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.402248 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.402246 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-workload-socket\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.402513 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.402266 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-token\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.402513 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.402285 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-data\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.402513 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.402340 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4a3115-4891-4117-82cd-7cbd804cef03-istiod-ca-cert\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.402513 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.402365 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvkpc\" (UniqueName: \"kubernetes.io/projected/fc4a3115-4891-4117-82cd-7cbd804cef03-kube-api-access-xvkpc\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.402513 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.402399 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-workload-certs\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.402778 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.402520 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-credential-socket\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.402778 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.402555 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-envoy\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.402778 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.402667 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-workload-socket\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.402778 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.402720 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-data\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.402989 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.402971 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-workload-certs\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.403053 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.403022 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-credential-socket\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.403182 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.403159 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4a3115-4891-4117-82cd-7cbd804cef03-istiod-ca-cert\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.405013 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.404991 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-podinfo\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.405099 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.405032 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-envoy\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.412571 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.412541 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-token\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.412856 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.412834 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvkpc\" (UniqueName: \"kubernetes.io/projected/fc4a3115-4891-4117-82cd-7cbd804cef03-kube-api-access-xvkpc\") pod \"e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.516605 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.516517 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:04.657018 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.656981 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:04.805951 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.805910 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6"] Sep 10 16:21:04.807437 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:21:04.807409 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4a3115_4891_4117_82cd_7cbd804cef03.slice/crio-cbbf647bc75290fcd0cd4952902a720c39e8f6d33a25414c5715e5d4acddf7bc WatchSource:0}: Error finding container cbbf647bc75290fcd0cd4952902a720c39e8f6d33a25414c5715e5d4acddf7bc: Status 404 returned error can't find the container with id cbbf647bc75290fcd0cd4952902a720c39e8f6d33a25414c5715e5d4acddf7bc Sep 10 16:21:04.809807 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.809791 2541 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 16:21:04.823016 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:04.822976 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" event={"ID":"fc4a3115-4891-4117-82cd-7cbd804cef03","Type":"ContainerStarted","Data":"cbbf647bc75290fcd0cd4952902a720c39e8f6d33a25414c5715e5d4acddf7bc"} Sep 10 16:21:06.929628 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:06.929564 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:07.651071 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:07.651038 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:07.955798 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:07.955705 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:08.089981 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:08.089950 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a64cdb4-gkwv7"] Sep 10 16:21:08.096245 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:08.096212 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a64cdb4-gkwv7"] Sep 10 16:21:08.273031 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:08.272936 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:08.774526 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:08.774490 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:21:08.980217 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:08.980181 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e4d04b9-25b2-4e6b-b85a-2e574b2609f5" path="/var/lib/kubelet/pods/0e4d04b9-25b2-4e6b-b85a-2e574b2609f5/volumes" Sep 10 16:21:09.287189 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:09.287126 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:09.691438 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:09.691395 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:10.136660 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:10.136632 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:10.499260 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:10.499172 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:12.828436 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:12.828403 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:12.915625 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:12.915588 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:13.318669 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:13.318631 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:14.139070 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:14.139029 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:14.238443 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:14.238405 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:14.416464 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:14.416369 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:14.722146 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:14.722057 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:14.826721 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:14.826679 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:16.202587 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:16.202552 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:21:16.202864 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:16.202626 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:21:16.202864 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:16.202655 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:21:16.908705 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:16.908668 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" event={"ID":"fc4a3115-4891-4117-82cd-7cbd804cef03","Type":"ContainerStarted","Data":"22ca81cf3ca4e688ae852827a28e201ec0f2ce93383a5ba17a067d18e854ad2c"} Sep 10 16:21:16.938659 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:16.938605 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" podStartSLOduration=1.546340239 podStartE2EDuration="12.938585955s" totalImagesPullingTime="11.392245716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:21:16.937439446 +0000 UTC m=+1176.460684643" watchObservedRunningTime="2026-09-10 16:21:16.938585955 +0000 UTC m=+1176.461831157" Sep 10 16:21:17.517671 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:17.517624 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:17.523079 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:17.523055 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:17.915866 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:17.915839 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:17.916874 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:17.916846 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:19.032345 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:19.032254 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:19.128466 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:19.128431 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:20.236205 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:20.236175 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:20.333937 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:20.333864 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:20.792372 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:20.792344 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:20.899009 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:20.898961 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:21.216889 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:21.216851 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:24.090615 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:24.090582 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:21:24.820783 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:24.820748 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:33.957505 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:33.957462 2541 patch_prober.go:28] interesting pod/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 16:21:33.957944 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:33.957546 2541 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" podUID="fc4a3115-4891-4117-82cd-7cbd804cef03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 16:21:35.029008 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:35.028973 2541 generic.go:356] "Generic (PLEG): container finished" podID="fc4a3115-4891-4117-82cd-7cbd804cef03" containerID="22ca81cf3ca4e688ae852827a28e201ec0f2ce93383a5ba17a067d18e854ad2c" exitCode=0 Sep 10 16:21:35.029577 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:35.029049 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" event={"ID":"fc4a3115-4891-4117-82cd-7cbd804cef03","Type":"ContainerDied","Data":"22ca81cf3ca4e688ae852827a28e201ec0f2ce93383a5ba17a067d18e854ad2c"} Sep 10 16:21:35.029577 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:35.029513 2541 scope.go:117] "RemoveContainer" containerID="22ca81cf3ca4e688ae852827a28e201ec0f2ce93383a5ba17a067d18e854ad2c" Sep 10 16:21:35.031737 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:35.031696 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:21:35.031854 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:35.031807 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:21:35.031934 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:35.031860 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:21:35.344581 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:35.344496 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:36.037142 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:36.037110 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" event={"ID":"fc4a3115-4891-4117-82cd-7cbd804cef03","Type":"ContainerStarted","Data":"566f008a74d420e5af72e431dbf5b2f34ffdac6eee9df6622e9a29a753bf1769"} Sep 10 16:21:36.517087 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:36.517051 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:36.517342 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:36.517292 2541 patch_prober.go:28] interesting pod/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 10 16:21:36.517400 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:36.517366 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" podUID="fc4a3115-4891-4117-82cd-7cbd804cef03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 10 16:21:37.517062 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:37.517023 2541 patch_prober.go:28] interesting pod/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 10 16:21:37.517507 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:37.517088 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" podUID="fc4a3115-4891-4117-82cd-7cbd804cef03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 10 16:21:37.681992 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:37.681958 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e28e849-nx22w"] Sep 10 16:21:37.686623 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:37.686594 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e28e849-nx22w" Sep 10 16:21:37.689875 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:37.689845 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:21:37.690524 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:37.690502 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l2dj2\"" Sep 10 16:21:37.706007 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:37.705981 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e28e849-nx22w"] Sep 10 16:21:37.724945 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:37.724909 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9909cbb-6106-48e7-8744-e939600723d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0e28e849-nx22w\" (UID: \"f9909cbb-6106-48e7-8744-e939600723d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e28e849-nx22w" Sep 10 16:21:37.826070 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:37.825966 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9909cbb-6106-48e7-8744-e939600723d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0e28e849-nx22w\" (UID: \"f9909cbb-6106-48e7-8744-e939600723d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e28e849-nx22w" Sep 10 16:21:37.826677 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:37.826653 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9909cbb-6106-48e7-8744-e939600723d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0e28e849-nx22w\" (UID: \"f9909cbb-6106-48e7-8744-e939600723d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e28e849-nx22w" Sep 10 16:21:38.007546 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:38.007503 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e28e849-nx22w" Sep 10 16:21:38.293932 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:38.293889 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e28e849-nx22w"] Sep 10 16:21:38.310479 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:21:38.310447 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9909cbb_6106_48e7_8744_e939600723d0.slice/crio-44a8c06c5cd6c476f1ff510620b8cdc51d74e4a79ea6934669fd0c95b74561a6 WatchSource:0}: Error finding container 44a8c06c5cd6c476f1ff510620b8cdc51d74e4a79ea6934669fd0c95b74561a6: Status 404 returned error can't find the container with id 44a8c06c5cd6c476f1ff510620b8cdc51d74e4a79ea6934669fd0c95b74561a6 Sep 10 16:21:38.517669 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:38.517633 2541 patch_prober.go:28] interesting pod/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 10 16:21:38.518051 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:38.517694 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" podUID="fc4a3115-4891-4117-82cd-7cbd804cef03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 10 16:21:39.060835 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:39.060734 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e28e849-nx22w" event={"ID":"f9909cbb-6106-48e7-8744-e939600723d0","Type":"ContainerStarted","Data":"1a4f589e2117724acb0787fd540c2a2e969680b8592c93fc133ecb2b92b6f2bd"} Sep 10 16:21:39.060835 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:39.060787 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e28e849-nx22w" event={"ID":"f9909cbb-6106-48e7-8744-e939600723d0","Type":"ContainerStarted","Data":"44a8c06c5cd6c476f1ff510620b8cdc51d74e4a79ea6934669fd0c95b74561a6"} Sep 10 16:21:39.517357 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:39.517293 2541 patch_prober.go:28] interesting pod/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 10 16:21:39.517538 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:39.517385 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" podUID="fc4a3115-4891-4117-82cd-7cbd804cef03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 10 16:21:40.068503 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:40.068470 2541 generic.go:356] "Generic (PLEG): container finished" podID="f9909cbb-6106-48e7-8744-e939600723d0" containerID="1a4f589e2117724acb0787fd540c2a2e969680b8592c93fc133ecb2b92b6f2bd" exitCode=0 Sep 10 16:21:40.068898 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:40.068515 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e28e849-nx22w" event={"ID":"f9909cbb-6106-48e7-8744-e939600723d0","Type":"ContainerDied","Data":"1a4f589e2117724acb0787fd540c2a2e969680b8592c93fc133ecb2b92b6f2bd"} Sep 10 16:21:40.517647 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:40.517608 2541 patch_prober.go:28] interesting pod/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 10 16:21:40.517827 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:40.517665 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" podUID="fc4a3115-4891-4117-82cd-7cbd804cef03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 10 16:21:41.045067 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:41.045034 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/1.log" Sep 10 16:21:41.046399 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:41.046376 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/1.log" Sep 10 16:21:41.051105 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:41.051080 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4c6j_2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd/ovn-acl-logging/0.log" Sep 10 16:21:41.052423 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:41.052403 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4c6j_2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd/ovn-acl-logging/0.log" Sep 10 16:21:41.269817 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:41.269787 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e28e849-nx22w" Sep 10 16:21:41.360291 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:41.360190 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9909cbb-6106-48e7-8744-e939600723d0-maas-api-service-ca\") pod \"f9909cbb-6106-48e7-8744-e939600723d0\" (UID: \"f9909cbb-6106-48e7-8744-e939600723d0\") " Sep 10 16:21:41.360640 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:41.360612 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9909cbb-6106-48e7-8744-e939600723d0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f9909cbb-6106-48e7-8744-e939600723d0" (UID: "f9909cbb-6106-48e7-8744-e939600723d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:21:41.461405 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:41.461368 2541 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9909cbb-6106-48e7-8744-e939600723d0-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:21:41.712416 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:41.712376 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:41.712895 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:41.712827 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:41.713971 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:41.713947 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:21:42.081278 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:42.081189 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e28e849-nx22w" Sep 10 16:21:42.081278 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:42.081221 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e28e849-nx22w" event={"ID":"f9909cbb-6106-48e7-8744-e939600723d0","Type":"ContainerDied","Data":"44a8c06c5cd6c476f1ff510620b8cdc51d74e4a79ea6934669fd0c95b74561a6"} Sep 10 16:21:42.081278 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:42.081276 2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44a8c06c5cd6c476f1ff510620b8cdc51d74e4a79ea6934669fd0c95b74561a6" Sep 10 16:21:42.851236 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:42.851205 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:44.749505 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:44.749455 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:44.832542 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:44.832512 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:44.930439 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:44.930402 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:44.987190 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:44.987160 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e28e849-nx22w"] Sep 10 16:21:44.991525 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:44.991492 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e28e849-nx22w"] Sep 10 16:21:45.032180 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:45.032091 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:45.138577 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:45.138525 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:45.858686 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:45.858651 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:45.979446 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:45.979415 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:46.109359 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:46.109252 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:46.979436 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:46.979400 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9909cbb-6106-48e7-8744-e939600723d0" path="/var/lib/kubelet/pods/f9909cbb-6106-48e7-8744-e939600723d0/volumes" Sep 10 16:21:47.492159 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:47.492124 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:48.261864 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:48.261826 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:21:49.047335 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:49.047289 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:53.900539 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:53.900492 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:54.011529 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:54.011485 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:54.108223 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:54.108186 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:54.206140 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:54.206064 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:54.602575 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:21:54.602494 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:02.652279 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:02.652242 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:22:17.166226 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:17.166190 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:22:20.532987 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:20.532949 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:22.077392 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:22.077353 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:24.058759 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:24.058717 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:27.319752 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:27.319721 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:27.670496 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:27.670453 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:27.805737 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:27.805674 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:27.952572 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:27.952491 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:29.316235 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:29.316201 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9"] Sep 10 16:22:29.353714 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:29.353681 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:30.362223 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:30.362184 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:30.622404 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:30.622372 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:32.062516 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:32.062477 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:22:37.585766 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:37.585730 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:38.293842 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:38.293806 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:38.387941 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:38.387906 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:38.491003 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:38.490963 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:38.585439 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:38.585339 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:38.939644 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:38.939609 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:40.619370 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:40.619337 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:41.345180 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:41.345137 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:41.391215 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:41.391176 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5"] Sep 10 16:22:41.391518 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:41.391489 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5" podUID="c95fb447-93f6-49ba-992c-276c2303d1f4" containerName="payload-processing" containerID="cri-o://2812a563f7b06b6a9baf5c553f6fa05bf05b0e34a5987879418a68a442b650c4" gracePeriod=30 Sep 10 16:22:41.474689 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:41.474656 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:41.565167 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:41.565122 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:41.641905 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:41.641872 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5" Sep 10 16:22:41.649051 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:41.649017 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:41.725427 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:41.725388 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95fb447-93f6-49ba-992c-276c2303d1f4-plugins-config-volume\") pod \"c95fb447-93f6-49ba-992c-276c2303d1f4\" (UID: \"c95fb447-93f6-49ba-992c-276c2303d1f4\") " Sep 10 16:22:41.725659 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:41.725488 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9b7x\" (UniqueName: \"kubernetes.io/projected/c95fb447-93f6-49ba-992c-276c2303d1f4-kube-api-access-m9b7x\") pod \"c95fb447-93f6-49ba-992c-276c2303d1f4\" (UID: \"c95fb447-93f6-49ba-992c-276c2303d1f4\") " Sep 10 16:22:41.725953 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:41.725920 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c95fb447-93f6-49ba-992c-276c2303d1f4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c95fb447-93f6-49ba-992c-276c2303d1f4" (UID: "c95fb447-93f6-49ba-992c-276c2303d1f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:22:41.728383 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:41.728350 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c95fb447-93f6-49ba-992c-276c2303d1f4-kube-api-access-m9b7x" (OuterVolumeSpecName: "kube-api-access-m9b7x") pod "c95fb447-93f6-49ba-992c-276c2303d1f4" (UID: "c95fb447-93f6-49ba-992c-276c2303d1f4"). InnerVolumeSpecName "kube-api-access-m9b7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:22:41.746936 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:41.746906 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:41.826546 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:41.826508 2541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95fb447-93f6-49ba-992c-276c2303d1f4-plugins-config-volume\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:22:41.826546 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:41.826542 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9b7x\" (UniqueName: \"kubernetes.io/projected/c95fb447-93f6-49ba-992c-276c2303d1f4-kube-api-access-m9b7x\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:22:42.027545 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:42.027509 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:42.450413 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:42.450377 2541 generic.go:356] "Generic (PLEG): container finished" podID="c95fb447-93f6-49ba-992c-276c2303d1f4" containerID="2812a563f7b06b6a9baf5c553f6fa05bf05b0e34a5987879418a68a442b650c4" exitCode=0 Sep 10 16:22:42.450595 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:42.450432 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5" Sep 10 16:22:42.450595 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:42.450466 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5" event={"ID":"c95fb447-93f6-49ba-992c-276c2303d1f4","Type":"ContainerDied","Data":"2812a563f7b06b6a9baf5c553f6fa05bf05b0e34a5987879418a68a442b650c4"} Sep 10 16:22:42.450595 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:42.450525 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5" event={"ID":"c95fb447-93f6-49ba-992c-276c2303d1f4","Type":"ContainerDied","Data":"c101c646f2db56d76a7f51534ee50b95f43fcabae2faf18c797b0286f497ad35"} Sep 10 16:22:42.450595 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:42.450550 2541 scope.go:117] "RemoveContainer" containerID="2812a563f7b06b6a9baf5c553f6fa05bf05b0e34a5987879418a68a442b650c4" Sep 10 16:22:42.460880 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:42.460859 2541 scope.go:117] "RemoveContainer" containerID="2812a563f7b06b6a9baf5c553f6fa05bf05b0e34a5987879418a68a442b650c4" Sep 10 16:22:42.461240 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:22:42.461201 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2812a563f7b06b6a9baf5c553f6fa05bf05b0e34a5987879418a68a442b650c4\": container with ID starting with 2812a563f7b06b6a9baf5c553f6fa05bf05b0e34a5987879418a68a442b650c4 not found: ID does not exist" containerID="2812a563f7b06b6a9baf5c553f6fa05bf05b0e34a5987879418a68a442b650c4" Sep 10 16:22:42.461353 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:42.461248 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2812a563f7b06b6a9baf5c553f6fa05bf05b0e34a5987879418a68a442b650c4"} err="failed to get container status \"2812a563f7b06b6a9baf5c553f6fa05bf05b0e34a5987879418a68a442b650c4\": rpc error: code = NotFound desc = could not find container \"2812a563f7b06b6a9baf5c553f6fa05bf05b0e34a5987879418a68a442b650c4\": container with ID starting with 2812a563f7b06b6a9baf5c553f6fa05bf05b0e34a5987879418a68a442b650c4 not found: ID does not exist" Sep 10 16:22:42.476331 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:42.476279 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5"] Sep 10 16:22:42.486845 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:42.486816 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-37c805-59bd77f68f-klcd5"] Sep 10 16:22:42.979283 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:42.979252 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c95fb447-93f6-49ba-992c-276c2303d1f4" path="/var/lib/kubelet/pods/c95fb447-93f6-49ba-992c-276c2303d1f4/volumes" Sep 10 16:22:43.820313 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:43.820267 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:43.861406 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:43.861367 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:22:44.336958 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:44.336926 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" podUID="34fc915d-d815-4ec4-a521-051f2d697fce" containerName="main" containerID="cri-o://fa0eb623f13cb15f56f779c1e0402135b6d4200fb71646eabdbd97ba6f4e46c6" gracePeriod=45 Sep 10 16:22:44.465252 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:44.465215 2541 generic.go:356] "Generic (PLEG): container finished" podID="34fc915d-d815-4ec4-a521-051f2d697fce" containerID="fa0eb623f13cb15f56f779c1e0402135b6d4200fb71646eabdbd97ba6f4e46c6" exitCode=0 Sep 10 16:22:44.465409 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:44.465284 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" event={"ID":"34fc915d-d815-4ec4-a521-051f2d697fce","Type":"ContainerDied","Data":"fa0eb623f13cb15f56f779c1e0402135b6d4200fb71646eabdbd97ba6f4e46c6"} Sep 10 16:22:44.591618 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:44.591559 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:22:44.756573 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:44.756535 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-tmp-dir\") pod \"34fc915d-d815-4ec4-a521-051f2d697fce\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " Sep 10 16:22:44.756573 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:44.756577 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-dshm\") pod \"34fc915d-d815-4ec4-a521-051f2d697fce\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " Sep 10 16:22:44.756834 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:44.756609 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcs4j\" (UniqueName: \"kubernetes.io/projected/34fc915d-d815-4ec4-a521-051f2d697fce-kube-api-access-xcs4j\") pod \"34fc915d-d815-4ec4-a521-051f2d697fce\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " Sep 10 16:22:44.756834 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:44.756665 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34fc915d-d815-4ec4-a521-051f2d697fce-tls-certs\") pod \"34fc915d-d815-4ec4-a521-051f2d697fce\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " Sep 10 16:22:44.756834 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:44.756697 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-home\") pod \"34fc915d-d815-4ec4-a521-051f2d697fce\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " Sep 10 16:22:44.756834 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:44.756734 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-model-cache\") pod \"34fc915d-d815-4ec4-a521-051f2d697fce\" (UID: \"34fc915d-d815-4ec4-a521-051f2d697fce\") " Sep 10 16:22:44.757054 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:44.756837 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "34fc915d-d815-4ec4-a521-051f2d697fce" (UID: "34fc915d-d815-4ec4-a521-051f2d697fce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:22:44.757054 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:44.756993 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-home" (OuterVolumeSpecName: "home") pod "34fc915d-d815-4ec4-a521-051f2d697fce" (UID: "34fc915d-d815-4ec4-a521-051f2d697fce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:22:44.757054 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:44.757033 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-model-cache" (OuterVolumeSpecName: "model-cache") pod "34fc915d-d815-4ec4-a521-051f2d697fce" (UID: "34fc915d-d815-4ec4-a521-051f2d697fce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:22:44.757202 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:44.757045 2541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-tmp-dir\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:22:44.757202 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:44.757084 2541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-home\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:22:44.759099 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:44.759073 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-dshm" (OuterVolumeSpecName: "dshm") pod "34fc915d-d815-4ec4-a521-051f2d697fce" (UID: "34fc915d-d815-4ec4-a521-051f2d697fce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:22:44.759204 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:44.759118 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34fc915d-d815-4ec4-a521-051f2d697fce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34fc915d-d815-4ec4-a521-051f2d697fce" (UID: "34fc915d-d815-4ec4-a521-051f2d697fce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:22:44.759247 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:44.759204 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34fc915d-d815-4ec4-a521-051f2d697fce-kube-api-access-xcs4j" (OuterVolumeSpecName: "kube-api-access-xcs4j") pod "34fc915d-d815-4ec4-a521-051f2d697fce" (UID: "34fc915d-d815-4ec4-a521-051f2d697fce"). InnerVolumeSpecName "kube-api-access-xcs4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:22:44.857793 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:44.857708 2541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34fc915d-d815-4ec4-a521-051f2d697fce-tls-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:22:44.857793 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:44.857739 2541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-model-cache\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:22:44.857793 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:44.857747 2541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34fc915d-d815-4ec4-a521-051f2d697fce-dshm\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:22:44.857793 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:44.857755 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcs4j\" (UniqueName: \"kubernetes.io/projected/34fc915d-d815-4ec4-a521-051f2d697fce-kube-api-access-xcs4j\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:22:45.472308 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:45.472275 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" event={"ID":"34fc915d-d815-4ec4-a521-051f2d697fce","Type":"ContainerDied","Data":"8b3b3e72e4ce2f77e6d8097513b4e394147af0906aae852517dd730a396349d3"} Sep 10 16:22:45.472770 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:45.472324 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9" Sep 10 16:22:45.472770 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:45.472348 2541 scope.go:117] "RemoveContainer" containerID="fa0eb623f13cb15f56f779c1e0402135b6d4200fb71646eabdbd97ba6f4e46c6" Sep 10 16:22:45.496066 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:45.496019 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9"] Sep 10 16:22:45.503768 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:45.503737 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-37c805/test-model-37c805-kserve-75465bbcd9-t9nm9"] Sep 10 16:22:46.110717 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:46.110682 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:46.224514 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:46.224483 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:46.326637 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:46.326603 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6"] Sep 10 16:22:46.326918 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:46.326886 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" podUID="fc4a3115-4891-4117-82cd-7cbd804cef03" containerName="istio-proxy" containerID="cri-o://566f008a74d420e5af72e431dbf5b2f34ffdac6eee9df6622e9a29a753bf1769" gracePeriod=30 Sep 10 16:22:46.353786 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:46.353755 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:46.834649 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:46.834614 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:46.981701 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:46.981663 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34fc915d-d815-4ec4-a521-051f2d697fce" path="/var/lib/kubelet/pods/34fc915d-d815-4ec4-a521-051f2d697fce/volumes" Sep 10 16:22:47.317251 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:47.317167 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:47.437267 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:47.437210 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:48.399558 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:48.399527 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:48.493102 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:48.493069 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:48.594612 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:48.594580 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:49.703503 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:49.703471 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:49.801229 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:49.801194 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:50.024738 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:50.024661 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:50.546654 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:50.546606 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:51.512311 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.512254 2541 generic.go:356] "Generic (PLEG): container finished" podID="fc4a3115-4891-4117-82cd-7cbd804cef03" containerID="566f008a74d420e5af72e431dbf5b2f34ffdac6eee9df6622e9a29a753bf1769" exitCode=0 Sep 10 16:22:51.512704 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.512340 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" event={"ID":"fc4a3115-4891-4117-82cd-7cbd804cef03","Type":"ContainerDied","Data":"566f008a74d420e5af72e431dbf5b2f34ffdac6eee9df6622e9a29a753bf1769"} Sep 10 16:22:51.512704 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.512404 2541 scope.go:117] "RemoveContainer" containerID="22ca81cf3ca4e688ae852827a28e201ec0f2ce93383a5ba17a067d18e854ad2c" Sep 10 16:22:51.585313 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.585271 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:22:51.724807 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.724783 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-data\") pod \"fc4a3115-4891-4117-82cd-7cbd804cef03\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " Sep 10 16:22:51.724972 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.724821 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-credential-socket\") pod \"fc4a3115-4891-4117-82cd-7cbd804cef03\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " Sep 10 16:22:51.724972 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.724885 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-podinfo\") pod \"fc4a3115-4891-4117-82cd-7cbd804cef03\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " Sep 10 16:22:51.724972 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.724909 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4a3115-4891-4117-82cd-7cbd804cef03-istiod-ca-cert\") pod \"fc4a3115-4891-4117-82cd-7cbd804cef03\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " Sep 10 16:22:51.724972 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.724956 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvkpc\" (UniqueName: \"kubernetes.io/projected/fc4a3115-4891-4117-82cd-7cbd804cef03-kube-api-access-xvkpc\") pod \"fc4a3115-4891-4117-82cd-7cbd804cef03\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " Sep 10 16:22:51.725170 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.724997 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-workload-socket\") pod \"fc4a3115-4891-4117-82cd-7cbd804cef03\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " Sep 10 16:22:51.725170 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.725041 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-envoy\") pod \"fc4a3115-4891-4117-82cd-7cbd804cef03\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " Sep 10 16:22:51.725170 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.725062 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-token\") pod \"fc4a3115-4891-4117-82cd-7cbd804cef03\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " Sep 10 16:22:51.725170 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.725099 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-workload-certs\") pod \"fc4a3115-4891-4117-82cd-7cbd804cef03\" (UID: \"fc4a3115-4891-4117-82cd-7cbd804cef03\") " Sep 10 16:22:51.725170 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.725105 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-data" (OuterVolumeSpecName: "istio-data") pod "fc4a3115-4891-4117-82cd-7cbd804cef03" (UID: "fc4a3115-4891-4117-82cd-7cbd804cef03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:22:51.725444 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.725180 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fc4a3115-4891-4117-82cd-7cbd804cef03" (UID: "fc4a3115-4891-4117-82cd-7cbd804cef03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:22:51.725444 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.725355 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fc4a3115-4891-4117-82cd-7cbd804cef03" (UID: "fc4a3115-4891-4117-82cd-7cbd804cef03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:22:51.725574 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.725460 2541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-workload-socket\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:22:51.725574 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.725483 2541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-data\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:22:51.725574 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.725498 2541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-credential-socket\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:22:51.725574 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.725457 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fc4a3115-4891-4117-82cd-7cbd804cef03" (UID: "fc4a3115-4891-4117-82cd-7cbd804cef03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:22:51.725574 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.725562 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc4a3115-4891-4117-82cd-7cbd804cef03-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fc4a3115-4891-4117-82cd-7cbd804cef03" (UID: "fc4a3115-4891-4117-82cd-7cbd804cef03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:22:51.727456 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.727433 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-token" (OuterVolumeSpecName: "istio-token") pod "fc4a3115-4891-4117-82cd-7cbd804cef03" (UID: "fc4a3115-4891-4117-82cd-7cbd804cef03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:22:51.727456 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.727433 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fc4a3115-4891-4117-82cd-7cbd804cef03" (UID: "fc4a3115-4891-4117-82cd-7cbd804cef03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:22:51.727599 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.727468 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fc4a3115-4891-4117-82cd-7cbd804cef03" (UID: "fc4a3115-4891-4117-82cd-7cbd804cef03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:22:51.727599 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.727473 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4a3115-4891-4117-82cd-7cbd804cef03-kube-api-access-xvkpc" (OuterVolumeSpecName: "kube-api-access-xvkpc") pod "fc4a3115-4891-4117-82cd-7cbd804cef03" (UID: "fc4a3115-4891-4117-82cd-7cbd804cef03"). InnerVolumeSpecName "kube-api-access-xvkpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:22:51.826727 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.826694 2541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-podinfo\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:22:51.826727 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.826726 2541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4a3115-4891-4117-82cd-7cbd804cef03-istiod-ca-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:22:51.826727 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.826735 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvkpc\" (UniqueName: \"kubernetes.io/projected/fc4a3115-4891-4117-82cd-7cbd804cef03-kube-api-access-xvkpc\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:22:51.826957 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.826747 2541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-envoy\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:22:51.826957 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.826755 2541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4a3115-4891-4117-82cd-7cbd804cef03-istio-token\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:22:51.826957 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:51.826762 2541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3115-4891-4117-82cd-7cbd804cef03-workload-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:22:52.266238 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:52.266202 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:52.376583 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:52.376549 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:52.477629 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:52.477597 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:52.522814 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:52.522737 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" event={"ID":"fc4a3115-4891-4117-82cd-7cbd804cef03","Type":"ContainerDied","Data":"cbbf647bc75290fcd0cd4952902a720c39e8f6d33a25414c5715e5d4acddf7bc"} Sep 10 16:22:52.522814 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:52.522795 2541 scope.go:117] "RemoveContainer" containerID="566f008a74d420e5af72e431dbf5b2f34ffdac6eee9df6622e9a29a753bf1769" Sep 10 16:22:52.523327 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:52.522751 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6" Sep 10 16:22:52.562217 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:52.562185 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6"] Sep 10 16:22:52.571652 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:52.571629 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-555853-openshift-default-6bdf47bfb8-7r4q6"] Sep 10 16:22:52.979898 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:22:52.979863 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc4a3115-4891-4117-82cd-7cbd804cef03" path="/var/lib/kubelet/pods/fc4a3115-4891-4117-82cd-7cbd804cef03/volumes" Sep 10 16:23:01.105398 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.105363 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:01.152812 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.152780 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m"] Sep 10 16:23:01.153018 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.152980 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m" podUID="8aae576e-3c67-4e21-95af-9895a2a305cf" containerName="payload-processing" containerID="cri-o://9ca234dcb0c2169ce3bb01db5f0d7358a58551dde0d3cee8f7ed372fa98dc36c" gracePeriod=30 Sep 10 16:23:01.164258 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.164112 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn"] Sep 10 16:23:01.164471 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.164444 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn" podUID="dd94c4ff-742e-4143-b72f-cbbade05817e" containerName="payload-pre-processing" containerID="cri-o://378c835f1d11f27b8eda363b10d067b54608d54ae53388a8b29b0f6082f539f7" gracePeriod=30 Sep 10 16:23:01.212462 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.212431 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:01.323679 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.323649 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:01.414689 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.414655 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:01.460716 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.460682 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m" Sep 10 16:23:01.463486 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.463471 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn" Sep 10 16:23:01.500823 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.500793 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7drfk\" (UniqueName: \"kubernetes.io/projected/8aae576e-3c67-4e21-95af-9895a2a305cf-kube-api-access-7drfk\") pod \"8aae576e-3c67-4e21-95af-9895a2a305cf\" (UID: \"8aae576e-3c67-4e21-95af-9895a2a305cf\") " Sep 10 16:23:01.501031 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.500836 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz589\" (UniqueName: \"kubernetes.io/projected/dd94c4ff-742e-4143-b72f-cbbade05817e-kube-api-access-jz589\") pod \"dd94c4ff-742e-4143-b72f-cbbade05817e\" (UID: \"dd94c4ff-742e-4143-b72f-cbbade05817e\") " Sep 10 16:23:01.501031 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.500904 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd94c4ff-742e-4143-b72f-cbbade05817e-plugins-config-volume\") pod \"dd94c4ff-742e-4143-b72f-cbbade05817e\" (UID: \"dd94c4ff-742e-4143-b72f-cbbade05817e\") " Sep 10 16:23:01.501031 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.500958 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aae576e-3c67-4e21-95af-9895a2a305cf-plugins-config-volume\") pod \"8aae576e-3c67-4e21-95af-9895a2a305cf\" (UID: \"8aae576e-3c67-4e21-95af-9895a2a305cf\") " Sep 10 16:23:01.501387 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.501358 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aae576e-3c67-4e21-95af-9895a2a305cf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8aae576e-3c67-4e21-95af-9895a2a305cf" (UID: "8aae576e-3c67-4e21-95af-9895a2a305cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:23:01.501387 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.501368 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd94c4ff-742e-4143-b72f-cbbade05817e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dd94c4ff-742e-4143-b72f-cbbade05817e" (UID: "dd94c4ff-742e-4143-b72f-cbbade05817e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:23:01.503235 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.503207 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd94c4ff-742e-4143-b72f-cbbade05817e-kube-api-access-jz589" (OuterVolumeSpecName: "kube-api-access-jz589") pod "dd94c4ff-742e-4143-b72f-cbbade05817e" (UID: "dd94c4ff-742e-4143-b72f-cbbade05817e"). InnerVolumeSpecName "kube-api-access-jz589". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:23:01.503371 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.503285 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aae576e-3c67-4e21-95af-9895a2a305cf-kube-api-access-7drfk" (OuterVolumeSpecName: "kube-api-access-7drfk") pod "8aae576e-3c67-4e21-95af-9895a2a305cf" (UID: "8aae576e-3c67-4e21-95af-9895a2a305cf"). InnerVolumeSpecName "kube-api-access-7drfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:23:01.578003 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.577967 2541 generic.go:356] "Generic (PLEG): container finished" podID="8aae576e-3c67-4e21-95af-9895a2a305cf" containerID="9ca234dcb0c2169ce3bb01db5f0d7358a58551dde0d3cee8f7ed372fa98dc36c" exitCode=0 Sep 10 16:23:01.578197 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.578019 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m" Sep 10 16:23:01.578197 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.578044 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m" event={"ID":"8aae576e-3c67-4e21-95af-9895a2a305cf","Type":"ContainerDied","Data":"9ca234dcb0c2169ce3bb01db5f0d7358a58551dde0d3cee8f7ed372fa98dc36c"} Sep 10 16:23:01.578197 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.578098 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m" event={"ID":"8aae576e-3c67-4e21-95af-9895a2a305cf","Type":"ContainerDied","Data":"9b76778cdd611fe3c12c89183375abc881a7e4698f953937e16ebce57aa2bf00"} Sep 10 16:23:01.578197 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.578131 2541 scope.go:117] "RemoveContainer" containerID="9ca234dcb0c2169ce3bb01db5f0d7358a58551dde0d3cee8f7ed372fa98dc36c" Sep 10 16:23:01.579607 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.579584 2541 generic.go:356] "Generic (PLEG): container finished" podID="dd94c4ff-742e-4143-b72f-cbbade05817e" containerID="378c835f1d11f27b8eda363b10d067b54608d54ae53388a8b29b0f6082f539f7" exitCode=0 Sep 10 16:23:01.579729 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.579628 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn" Sep 10 16:23:01.579729 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.579637 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn" event={"ID":"dd94c4ff-742e-4143-b72f-cbbade05817e","Type":"ContainerDied","Data":"378c835f1d11f27b8eda363b10d067b54608d54ae53388a8b29b0f6082f539f7"} Sep 10 16:23:01.579729 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.579677 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn" event={"ID":"dd94c4ff-742e-4143-b72f-cbbade05817e","Type":"ContainerDied","Data":"a3f2600efa4e9a34584193a106d5bce892a519c02b92fab92cac586792860f46"} Sep 10 16:23:01.588144 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.588127 2541 scope.go:117] "RemoveContainer" containerID="9ca234dcb0c2169ce3bb01db5f0d7358a58551dde0d3cee8f7ed372fa98dc36c" Sep 10 16:23:01.588535 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:23:01.588499 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca234dcb0c2169ce3bb01db5f0d7358a58551dde0d3cee8f7ed372fa98dc36c\": container with ID starting with 9ca234dcb0c2169ce3bb01db5f0d7358a58551dde0d3cee8f7ed372fa98dc36c not found: ID does not exist" containerID="9ca234dcb0c2169ce3bb01db5f0d7358a58551dde0d3cee8f7ed372fa98dc36c" Sep 10 16:23:01.588611 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.588532 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca234dcb0c2169ce3bb01db5f0d7358a58551dde0d3cee8f7ed372fa98dc36c"} err="failed to get container status \"9ca234dcb0c2169ce3bb01db5f0d7358a58551dde0d3cee8f7ed372fa98dc36c\": rpc error: code = NotFound desc = could not find container \"9ca234dcb0c2169ce3bb01db5f0d7358a58551dde0d3cee8f7ed372fa98dc36c\": container with ID starting with 9ca234dcb0c2169ce3bb01db5f0d7358a58551dde0d3cee8f7ed372fa98dc36c not found: ID does not exist" Sep 10 16:23:01.588611 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.588554 2541 scope.go:117] "RemoveContainer" containerID="378c835f1d11f27b8eda363b10d067b54608d54ae53388a8b29b0f6082f539f7" Sep 10 16:23:01.600609 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.600581 2541 scope.go:117] "RemoveContainer" containerID="378c835f1d11f27b8eda363b10d067b54608d54ae53388a8b29b0f6082f539f7" Sep 10 16:23:01.601482 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:23:01.601458 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"378c835f1d11f27b8eda363b10d067b54608d54ae53388a8b29b0f6082f539f7\": container with ID starting with 378c835f1d11f27b8eda363b10d067b54608d54ae53388a8b29b0f6082f539f7 not found: ID does not exist" containerID="378c835f1d11f27b8eda363b10d067b54608d54ae53388a8b29b0f6082f539f7" Sep 10 16:23:01.601574 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.601492 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"378c835f1d11f27b8eda363b10d067b54608d54ae53388a8b29b0f6082f539f7"} err="failed to get container status \"378c835f1d11f27b8eda363b10d067b54608d54ae53388a8b29b0f6082f539f7\": rpc error: code = NotFound desc = could not find container \"378c835f1d11f27b8eda363b10d067b54608d54ae53388a8b29b0f6082f539f7\": container with ID starting with 378c835f1d11f27b8eda363b10d067b54608d54ae53388a8b29b0f6082f539f7 not found: ID does not exist" Sep 10 16:23:01.601672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.601653 2541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aae576e-3c67-4e21-95af-9895a2a305cf-plugins-config-volume\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:23:01.601718 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.601676 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7drfk\" (UniqueName: \"kubernetes.io/projected/8aae576e-3c67-4e21-95af-9895a2a305cf-kube-api-access-7drfk\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:23:01.601718 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.601685 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz589\" (UniqueName: \"kubernetes.io/projected/dd94c4ff-742e-4143-b72f-cbbade05817e-kube-api-access-jz589\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:23:01.601718 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.601694 2541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd94c4ff-742e-4143-b72f-cbbade05817e-plugins-config-volume\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:23:01.618826 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.618801 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m"] Sep 10 16:23:01.622981 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.622949 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e22c72-768765654f-sbn8m"] Sep 10 16:23:01.650404 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.650342 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn"] Sep 10 16:23:01.652908 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.652887 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e22c72-67c684d9c4-k5lcn"] Sep 10 16:23:01.801450 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:01.801420 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:02.980047 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:02.980015 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aae576e-3c67-4e21-95af-9895a2a305cf" path="/var/lib/kubelet/pods/8aae576e-3c67-4e21-95af-9895a2a305cf/volumes" Sep 10 16:23:02.980448 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:02.980371 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd94c4ff-742e-4143-b72f-cbbade05817e" path="/var/lib/kubelet/pods/dd94c4ff-742e-4143-b72f-cbbade05817e/volumes" Sep 10 16:23:02.984841 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:02.984821 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:03.914622 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:03.914581 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:04.153942 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:04.153902 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:23:07.891039 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:07.891005 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:08.305015 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:08.304931 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:08.404360 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:08.404327 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:08.587464 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:08.587368 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:09.134706 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.134671 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:09.439836 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.439761 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f"] Sep 10 16:23:09.440282 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.440265 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c95fb447-93f6-49ba-992c-276c2303d1f4" containerName="payload-processing" Sep 10 16:23:09.440346 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.440286 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="c95fb447-93f6-49ba-992c-276c2303d1f4" containerName="payload-processing" Sep 10 16:23:09.440346 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.440312 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34fc915d-d815-4ec4-a521-051f2d697fce" containerName="main" Sep 10 16:23:09.440346 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.440319 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="34fc915d-d815-4ec4-a521-051f2d697fce" containerName="main" Sep 10 16:23:09.440346 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.440337 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9909cbb-6106-48e7-8744-e939600723d0" containerName="revoke-keys" Sep 10 16:23:09.440346 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.440343 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9909cbb-6106-48e7-8744-e939600723d0" containerName="revoke-keys" Sep 10 16:23:09.440519 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.440353 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc4a3115-4891-4117-82cd-7cbd804cef03" containerName="istio-proxy" Sep 10 16:23:09.440519 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.440358 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc4a3115-4891-4117-82cd-7cbd804cef03" containerName="istio-proxy" Sep 10 16:23:09.440519 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.440366 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd94c4ff-742e-4143-b72f-cbbade05817e" containerName="payload-pre-processing" Sep 10 16:23:09.440519 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.440371 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd94c4ff-742e-4143-b72f-cbbade05817e" containerName="payload-pre-processing" Sep 10 16:23:09.440519 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.440378 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aae576e-3c67-4e21-95af-9895a2a305cf" containerName="payload-processing" Sep 10 16:23:09.440519 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.440384 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aae576e-3c67-4e21-95af-9895a2a305cf" containerName="payload-processing" Sep 10 16:23:09.440519 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.440483 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc4a3115-4891-4117-82cd-7cbd804cef03" containerName="istio-proxy" Sep 10 16:23:09.440519 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.440494 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="c95fb447-93f6-49ba-992c-276c2303d1f4" containerName="payload-processing" Sep 10 16:23:09.440519 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.440502 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="34fc915d-d815-4ec4-a521-051f2d697fce" containerName="main" Sep 10 16:23:09.440519 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.440512 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd94c4ff-742e-4143-b72f-cbbade05817e" containerName="payload-pre-processing" Sep 10 16:23:09.440519 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.440521 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aae576e-3c67-4e21-95af-9895a2a305cf" containerName="payload-processing" Sep 10 16:23:09.440981 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.440528 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9909cbb-6106-48e7-8744-e939600723d0" containerName="revoke-keys" Sep 10 16:23:09.440981 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.440534 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc4a3115-4891-4117-82cd-7cbd804cef03" containerName="istio-proxy" Sep 10 16:23:09.443834 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.443809 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.446613 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.446592 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-219fd6-openshift-default-dockercfg-tkzwb\"" Sep 10 16:23:09.455865 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.455839 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-workload-certs\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.455991 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.455881 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m22ks\" (UniqueName: \"kubernetes.io/projected/74ab292d-0342-4278-943b-a85c78d351b9-kube-api-access-m22ks\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.455991 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.455992 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-istio-envoy\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.456116 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.456023 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-credential-socket\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.456116 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.456050 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ab292d-0342-4278-943b-a85c78d351b9-istio-token\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.456116 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.456075 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-workload-socket\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.456116 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.456101 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-istio-data\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.456344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.456162 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ab292d-0342-4278-943b-a85c78d351b9-istio-podinfo\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.456344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.456185 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ab292d-0342-4278-943b-a85c78d351b9-istiod-ca-cert\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.458672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.458651 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f"] Sep 10 16:23:09.489192 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.489147 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:09.557152 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.557116 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-istio-envoy\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.557152 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.557160 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-credential-socket\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.557439 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.557179 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ab292d-0342-4278-943b-a85c78d351b9-istio-token\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.557439 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.557205 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-workload-socket\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.557439 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.557232 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-istio-data\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.557439 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.557270 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ab292d-0342-4278-943b-a85c78d351b9-istio-podinfo\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.557439 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.557321 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ab292d-0342-4278-943b-a85c78d351b9-istiod-ca-cert\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.557439 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.557401 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-workload-certs\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.557439 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.557432 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m22ks\" (UniqueName: \"kubernetes.io/projected/74ab292d-0342-4278-943b-a85c78d351b9-kube-api-access-m22ks\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.557832 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.557678 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-workload-socket\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.557832 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.557684 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-istio-data\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.557952 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.557858 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-workload-certs\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.557952 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.557945 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-credential-socket\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.558196 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.558171 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ab292d-0342-4278-943b-a85c78d351b9-istiod-ca-cert\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.559865 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.559840 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-istio-envoy\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.560546 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.560522 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ab292d-0342-4278-943b-a85c78d351b9-istio-podinfo\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.566613 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.566592 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ab292d-0342-4278-943b-a85c78d351b9-istio-token\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.567005 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.566986 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m22ks\" (UniqueName: \"kubernetes.io/projected/74ab292d-0342-4278-943b-a85c78d351b9-kube-api-access-m22ks\") pod \"e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:09.756175 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:09.756087 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:10.040538 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:10.040510 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f"] Sep 10 16:23:10.042631 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:23:10.042598 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ab292d_0342_4278_943b_a85c78d351b9.slice/crio-b38e06de8a1726c8460815b890e65c094c9289cc6ea6582c1187a7d89fe5419c WatchSource:0}: Error finding container b38e06de8a1726c8460815b890e65c094c9289cc6ea6582c1187a7d89fe5419c: Status 404 returned error can't find the container with id b38e06de8a1726c8460815b890e65c094c9289cc6ea6582c1187a7d89fe5419c Sep 10 16:23:10.045293 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:10.045260 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:23:10.045416 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:10.045384 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:23:10.045461 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:10.045428 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:23:10.118750 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:10.118716 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:10.637390 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:10.637353 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" event={"ID":"74ab292d-0342-4278-943b-a85c78d351b9","Type":"ContainerStarted","Data":"f0af5e19adbee84bfda95c6f76140eb2a99b32dc85dc849e6e1234245b626134"} Sep 10 16:23:10.637841 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:10.637405 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" event={"ID":"74ab292d-0342-4278-943b-a85c78d351b9","Type":"ContainerStarted","Data":"b38e06de8a1726c8460815b890e65c094c9289cc6ea6582c1187a7d89fe5419c"} Sep 10 16:23:10.663831 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:10.663764 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" podStartSLOduration=1.663717118 podStartE2EDuration="1.663717118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:23:10.661770499 +0000 UTC m=+1290.185015699" watchObservedRunningTime="2026-09-10 16:23:10.663717118 +0000 UTC m=+1290.186962320" Sep 10 16:23:10.677532 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:10.677470 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:10.756669 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:10.756631 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:10.768166 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:10.768140 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:10.867232 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:10.867201 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:10.964160 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:10.964060 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:11.740423 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:11.740384 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:11.761378 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:11.761348 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:11.913475 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:11.913443 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:12.649407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:12.649376 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:12.650359 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:12.650343 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:12.894348 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:12.894284 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:12.995370 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:12.995258 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:13.512759 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:13.512718 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:14.247199 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:14.247167 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:14.351340 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:14.351287 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:14.477916 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:14.477866 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:14.818639 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:14.818610 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:14.923018 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:14.922984 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:17.854002 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:17.853964 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:23:18.586013 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:18.585981 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:18.716080 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:18.716041 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:18.900951 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:18.900913 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:19.027637 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:19.027596 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:20.539350 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:20.539275 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:20.641660 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:20.641628 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:21.292785 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:21.292750 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:24.033753 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:24.033722 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:27.650973 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:27.650934 2541 patch_prober.go:28] interesting pod/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 10 16:23:27.651416 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:27.651026 2541 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" podUID="74ab292d-0342-4278-943b-a85c78d351b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 10 16:23:27.747476 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:27.747401 2541 generic.go:356] "Generic (PLEG): container finished" podID="74ab292d-0342-4278-943b-a85c78d351b9" containerID="f0af5e19adbee84bfda95c6f76140eb2a99b32dc85dc849e6e1234245b626134" exitCode=0 Sep 10 16:23:27.747476 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:27.747439 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" event={"ID":"74ab292d-0342-4278-943b-a85c78d351b9","Type":"ContainerDied","Data":"f0af5e19adbee84bfda95c6f76140eb2a99b32dc85dc849e6e1234245b626134"} Sep 10 16:23:27.747812 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:27.747798 2541 scope.go:117] "RemoveContainer" containerID="f0af5e19adbee84bfda95c6f76140eb2a99b32dc85dc849e6e1234245b626134" Sep 10 16:23:27.748888 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:27.748860 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:23:27.748967 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:27.748949 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:23:27.749014 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:27.749001 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:23:28.755083 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:28.755048 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" event={"ID":"74ab292d-0342-4278-943b-a85c78d351b9","Type":"ContainerStarted","Data":"e34dbff76f2ff42e895050d4d0df9684660632b5f5d07f45a948d3e11561b17c"} Sep 10 16:23:28.756537 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:28.756516 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:28.756794 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:28.756765 2541 patch_prober.go:28] interesting pod/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 10 16:23:28.756882 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:28.756808 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" podUID="74ab292d-0342-4278-943b-a85c78d351b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 10 16:23:28.829114 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:28.829077 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:28.928459 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:28.928424 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:29.128482 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:29.128440 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:29.459502 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:29.459425 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:29.757622 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:29.757543 2541 patch_prober.go:28] interesting pod/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 10 16:23:29.757622 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:29.757599 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" podUID="74ab292d-0342-4278-943b-a85c78d351b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 10 16:23:29.770995 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:29.770955 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:23:29.862280 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:29.862243 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:30.757009 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:30.756973 2541 patch_prober.go:28] interesting pod/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 10 16:23:30.757195 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:30.757030 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" podUID="74ab292d-0342-4278-943b-a85c78d351b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 10 16:23:31.756537 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:31.756503 2541 patch_prober.go:28] interesting pod/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 10 16:23:31.756933 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:31.756561 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" podUID="74ab292d-0342-4278-943b-a85c78d351b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 10 16:23:32.417028 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:32.416957 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:32.519987 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:32.519958 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:32.756567 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:32.756483 2541 patch_prober.go:28] interesting pod/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 10 16:23:32.756567 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:32.756540 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" podUID="74ab292d-0342-4278-943b-a85c78d351b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 10 16:23:33.761161 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:33.761129 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:33.761552 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:33.761428 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:33.762113 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:33.762095 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:23:33.963847 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:33.963810 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:34.156533 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:34.156497 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:34.678705 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:34.678671 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:34.897644 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:34.897608 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:36.551955 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:36.551924 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:37.834271 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:37.834235 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:38.079650 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.079615 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:23:38.280180 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.280143 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:38.556218 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.556137 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp"] Sep 10 16:23:38.556546 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.556530 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc4a3115-4891-4117-82cd-7cbd804cef03" containerName="istio-proxy" Sep 10 16:23:38.556546 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.556546 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc4a3115-4891-4117-82cd-7cbd804cef03" containerName="istio-proxy" Sep 10 16:23:38.560853 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.560832 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.564047 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.564025 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-b73303-openshift-default-dockercfg-vtvjh\"" Sep 10 16:23:38.583672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.583641 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp"] Sep 10 16:23:38.640329 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.640271 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:38.647155 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.646643 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-workload-certs\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.647472 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.647439 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-envoy\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.647651 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.647634 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-credential-socket\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.647811 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.647797 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e1c048-296e-4512-98fe-085cc50d61b6-istiod-ca-cert\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.648000 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.647985 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-podinfo\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.648151 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.648135 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvkpb\" (UniqueName: \"kubernetes.io/projected/a0e1c048-296e-4512-98fe-085cc50d61b6-kube-api-access-tvkpb\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.648483 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.648451 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-workload-socket\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.648647 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.648615 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-token\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.648788 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.648762 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-data\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.750220 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.750187 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-workload-certs\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.750425 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.750231 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-envoy\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.750425 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.750250 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-credential-socket\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.750425 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.750268 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e1c048-296e-4512-98fe-085cc50d61b6-istiod-ca-cert\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.750425 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.750330 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-podinfo\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.750425 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.750352 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvkpb\" (UniqueName: \"kubernetes.io/projected/a0e1c048-296e-4512-98fe-085cc50d61b6-kube-api-access-tvkpb\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.750425 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.750386 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-workload-socket\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.750425 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.750408 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-token\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.750893 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.750430 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-data\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.750893 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.750747 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-credential-socket\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.750893 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.750805 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-workload-certs\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.750893 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.750831 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-workload-socket\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.751041 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.750948 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-data\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.751256 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.751238 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e1c048-296e-4512-98fe-085cc50d61b6-istiod-ca-cert\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.753388 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.753362 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-envoy\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.753576 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.753558 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-podinfo\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.762589 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.762569 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-token\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.762747 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.762729 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvkpb\" (UniqueName: \"kubernetes.io/projected/a0e1c048-296e-4512-98fe-085cc50d61b6-kube-api-access-tvkpb\") pod \"e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:38.872110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:38.872079 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:39.029874 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:39.029842 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:39.196347 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:39.196318 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp"] Sep 10 16:23:39.198226 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:23:39.198198 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e1c048_296e_4512_98fe_085cc50d61b6.slice/crio-a2bffbb80c3decd988dcbc3458f80fd45cafa953dd5e737b9e704b3853e7af74 WatchSource:0}: Error finding container a2bffbb80c3decd988dcbc3458f80fd45cafa953dd5e737b9e704b3853e7af74: Status 404 returned error can't find the container with id a2bffbb80c3decd988dcbc3458f80fd45cafa953dd5e737b9e704b3853e7af74 Sep 10 16:23:39.200294 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:39.200261 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:23:39.200385 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:39.200349 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:23:39.200425 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:39.200384 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:23:39.822473 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:39.822435 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" event={"ID":"a0e1c048-296e-4512-98fe-085cc50d61b6","Type":"ContainerStarted","Data":"d690a44ffc80a1951d24d671c594c622945dd661cdc73bcb63cbf95569280a3b"} Sep 10 16:23:39.822655 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:39.822499 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" event={"ID":"a0e1c048-296e-4512-98fe-085cc50d61b6","Type":"ContainerStarted","Data":"a2bffbb80c3decd988dcbc3458f80fd45cafa953dd5e737b9e704b3853e7af74"} Sep 10 16:23:39.872709 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:39.872682 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:39.891789 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:39.891740 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" podStartSLOduration=1.891724172 podStartE2EDuration="1.891724172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:23:39.889182158 +0000 UTC m=+1319.412427355" watchObservedRunningTime="2026-09-10 16:23:39.891724172 +0000 UTC m=+1319.414969369" Sep 10 16:23:40.878242 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:40.878211 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:41.837653 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:41.837625 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:41.838584 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:41.838565 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:41.895693 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:41.895654 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:41.999762 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:41.999723 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:43.212439 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:43.212407 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:43.314622 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:43.314584 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:43.413147 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:43.413117 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:43.796741 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:43.796709 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:43.904713 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:43.904682 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:47.556957 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:47.556921 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:23:48.540316 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:48.540260 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:55.242598 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:55.242564 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:57.177574 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:57.176661 2541 patch_prober.go:28] interesting pod/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": read tcp 10.132.0.2:56518->10.132.0.65:15021: read: connection reset by peer" start-of-body= Sep 10 16:23:57.177574 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:57.176744 2541 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" podUID="a0e1c048-296e-4512-98fe-085cc50d61b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": read tcp 10.132.0.2:56518->10.132.0.65:15021: read: connection reset by peer" Sep 10 16:23:57.428088 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:57.427999 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:57.940243 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:57.940207 2541 generic.go:356] "Generic (PLEG): container finished" podID="a0e1c048-296e-4512-98fe-085cc50d61b6" containerID="d690a44ffc80a1951d24d671c594c622945dd661cdc73bcb63cbf95569280a3b" exitCode=0 Sep 10 16:23:57.940436 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:57.940274 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" event={"ID":"a0e1c048-296e-4512-98fe-085cc50d61b6","Type":"ContainerDied","Data":"d690a44ffc80a1951d24d671c594c622945dd661cdc73bcb63cbf95569280a3b"} Sep 10 16:23:57.940701 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:57.940687 2541 scope.go:117] "RemoveContainer" containerID="d690a44ffc80a1951d24d671c594c622945dd661cdc73bcb63cbf95569280a3b" Sep 10 16:23:57.941724 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:57.941699 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:23:57.941774 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:57.941763 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:23:57.941808 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:57.941796 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:23:58.155768 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:58.155734 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:58.256852 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:58.256755 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:58.674106 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:58.674071 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:58.948231 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:58.948146 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" event={"ID":"a0e1c048-296e-4512-98fe-085cc50d61b6","Type":"ContainerStarted","Data":"b177a2345535b2a3ca4eadf863c9f867955da4da8da792b06cdd6e2bf33769df"} Sep 10 16:23:59.872688 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:59.872649 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:23:59.873138 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:59.872819 2541 patch_prober.go:28] interesting pod/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 10 16:23:59.873138 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:23:59.872884 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" podUID="a0e1c048-296e-4512-98fe-085cc50d61b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 10 16:24:00.872648 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:00.872614 2541 patch_prober.go:28] interesting pod/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 10 16:24:00.872849 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:00.872683 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" podUID="a0e1c048-296e-4512-98fe-085cc50d61b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 10 16:24:01.592351 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:01.592270 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:01.872628 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:01.872601 2541 patch_prober.go:28] interesting pod/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 10 16:24:01.872838 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:01.872659 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" podUID="a0e1c048-296e-4512-98fe-085cc50d61b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 10 16:24:02.873077 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:02.873036 2541 patch_prober.go:28] interesting pod/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 10 16:24:02.873519 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:02.873110 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" podUID="a0e1c048-296e-4512-98fe-085cc50d61b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 10 16:24:03.872751 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:03.872714 2541 patch_prober.go:28] interesting pod/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 10 16:24:03.872940 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:03.872772 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" podUID="a0e1c048-296e-4512-98fe-085cc50d61b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 10 16:24:04.798195 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:04.798157 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:04.868074 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:04.868040 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:04.876253 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:04.876226 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:24:04.876655 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:04.876637 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:24:04.877230 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:04.877212 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:24:05.207455 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:05.207422 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37c368a6-6h8np"] Sep 10 16:24:05.211016 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:05.211000 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37c368a6-6h8np" Sep 10 16:24:05.213099 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:05.213077 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:24:05.213203 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:05.213126 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l2dj2\"" Sep 10 16:24:05.219935 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:05.219901 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37c368a6-6h8np"] Sep 10 16:24:05.299955 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:05.299913 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a130e8c-d405-4805-8b87-91ec928b7b6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-37c368a6-6h8np\" (UID: \"1a130e8c-d405-4805-8b87-91ec928b7b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37c368a6-6h8np" Sep 10 16:24:05.400996 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:05.400960 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a130e8c-d405-4805-8b87-91ec928b7b6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-37c368a6-6h8np\" (UID: \"1a130e8c-d405-4805-8b87-91ec928b7b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37c368a6-6h8np" Sep 10 16:24:05.401616 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:05.401597 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a130e8c-d405-4805-8b87-91ec928b7b6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-37c368a6-6h8np\" (UID: \"1a130e8c-d405-4805-8b87-91ec928b7b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37c368a6-6h8np" Sep 10 16:24:05.523341 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:05.523220 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37c368a6-6h8np" Sep 10 16:24:05.797856 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:05.797820 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37c368a6-6h8np"] Sep 10 16:24:05.800086 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:24:05.800041 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a130e8c_d405_4805_8b87_91ec928b7b6e.slice/crio-a4e4d16d8845c1c9a92ec140fb483ea650e6c5584a2e88d70cfef31539619418 WatchSource:0}: Error finding container a4e4d16d8845c1c9a92ec140fb483ea650e6c5584a2e88d70cfef31539619418: Status 404 returned error can't find the container with id a4e4d16d8845c1c9a92ec140fb483ea650e6c5584a2e88d70cfef31539619418 Sep 10 16:24:05.868033 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:05.867998 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:05.991696 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:05.991662 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37c368a6-6h8np" event={"ID":"1a130e8c-d405-4805-8b87-91ec928b7b6e","Type":"ContainerStarted","Data":"a4e4d16d8845c1c9a92ec140fb483ea650e6c5584a2e88d70cfef31539619418"} Sep 10 16:24:06.733027 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:06.732991 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:06.999254 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:06.999222 2541 generic.go:356] "Generic (PLEG): container finished" podID="1a130e8c-d405-4805-8b87-91ec928b7b6e" containerID="0604a4b03ec3c251c9a6b2c397d75fd7560f37ae1d24335ee742848ef358114b" exitCode=0 Sep 10 16:24:06.999677 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:06.999325 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37c368a6-6h8np" event={"ID":"1a130e8c-d405-4805-8b87-91ec928b7b6e","Type":"ContainerDied","Data":"0604a4b03ec3c251c9a6b2c397d75fd7560f37ae1d24335ee742848ef358114b"} Sep 10 16:24:07.879550 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:07.879521 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:08.138097 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:08.138028 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37c368a6-6h8np" Sep 10 16:24:08.229996 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:08.229959 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a130e8c-d405-4805-8b87-91ec928b7b6e-maas-api-service-ca\") pod \"1a130e8c-d405-4805-8b87-91ec928b7b6e\" (UID: \"1a130e8c-d405-4805-8b87-91ec928b7b6e\") " Sep 10 16:24:08.230374 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:08.230352 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a130e8c-d405-4805-8b87-91ec928b7b6e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1a130e8c-d405-4805-8b87-91ec928b7b6e" (UID: "1a130e8c-d405-4805-8b87-91ec928b7b6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:24:08.331153 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:08.331114 2541 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a130e8c-d405-4805-8b87-91ec928b7b6e-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:24:09.012548 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:09.012520 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37c368a6-6h8np" Sep 10 16:24:09.012713 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:09.012561 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37c368a6-6h8np" event={"ID":"1a130e8c-d405-4805-8b87-91ec928b7b6e","Type":"ContainerDied","Data":"a4e4d16d8845c1c9a92ec140fb483ea650e6c5584a2e88d70cfef31539619418"} Sep 10 16:24:09.012713 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:09.012603 2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4e4d16d8845c1c9a92ec140fb483ea650e6c5584a2e88d70cfef31539619418" Sep 10 16:24:12.083952 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:12.083915 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:15.492677 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:15.492637 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:16.358026 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:16.357992 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:16.459847 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:16.459811 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:16.559414 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:16.559374 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:16.617928 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:16.617828 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37c368a6-6h8np"] Sep 10 16:24:16.620382 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:16.620360 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-37c368a6-6h8np"] Sep 10 16:24:16.659281 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:16.659246 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:16.980453 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:16.980409 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a130e8c-d405-4805-8b87-91ec928b7b6e" path="/var/lib/kubelet/pods/1a130e8c-d405-4805-8b87-91ec928b7b6e/volumes" Sep 10 16:24:17.026840 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:17.026807 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:19.555636 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:19.555602 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:24:23.415006 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:23.414965 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:23.510403 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:23.510368 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:23.742700 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:23.742604 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:23.982234 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:23.982203 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:24.110600 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:24.110514 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:24.202479 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:24.202442 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:24.354473 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:24.354442 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:24.725016 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:24.724977 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:31.118379 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:31.118345 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:31.244781 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:31.244747 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:31.381424 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:31.381395 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:31.627424 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:31.627396 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:34.068968 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:34.068937 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:24:38.357630 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:38.357570 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:38.473167 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:38.473135 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:39.408393 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:39.408353 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:39.838778 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:39.838678 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:40.130451 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:40.130417 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:42.792989 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:42.792955 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:42.892902 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:42.892860 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:43.988080 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:43.988041 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:44.087802 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:44.087771 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:44.323874 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:44.323793 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:44.547065 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:44.546994 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:44.645139 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:44.645102 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:48.416776 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.416738 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch"] Sep 10 16:24:48.417768 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.417739 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a130e8c-d405-4805-8b87-91ec928b7b6e" containerName="revoke-keys" Sep 10 16:24:48.417768 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.417770 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a130e8c-d405-4805-8b87-91ec928b7b6e" containerName="revoke-keys" Sep 10 16:24:48.418016 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.418000 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a130e8c-d405-4805-8b87-91ec928b7b6e" containerName="revoke-keys" Sep 10 16:24:48.422680 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.422554 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:48.425240 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.425221 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b73303\"/\"default-dockercfg-2jzpz\"" Sep 10 16:24:48.425371 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.425264 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b73303\"/\"kube-root-ca.crt\"" Sep 10 16:24:48.425591 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.425576 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b73303\"/\"openshift-service-ca.crt\"" Sep 10 16:24:48.425794 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.425770 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b73303\"/\"auth-test-model-b73303-kserve-self-signed-certs\"" Sep 10 16:24:48.433914 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.433892 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch"] Sep 10 16:24:48.505911 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.505876 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56276b2e-8546-488f-a878-4e3e071274b5-tls-certs\") pod \"auth-test-model-b73303-kserve-598f4bb9b7-lqsch\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:48.506087 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.505922 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-dshm\") pod \"auth-test-model-b73303-kserve-598f4bb9b7-lqsch\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:48.506087 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.505937 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-model-cache\") pod \"auth-test-model-b73303-kserve-598f4bb9b7-lqsch\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:48.506087 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.505968 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-home\") pod \"auth-test-model-b73303-kserve-598f4bb9b7-lqsch\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:48.506087 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.506008 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-tmp-dir\") pod \"auth-test-model-b73303-kserve-598f4bb9b7-lqsch\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:48.506087 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.506026 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dccgb\" (UniqueName: \"kubernetes.io/projected/56276b2e-8546-488f-a878-4e3e071274b5-kube-api-access-dccgb\") pod \"auth-test-model-b73303-kserve-598f4bb9b7-lqsch\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:48.607540 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.607508 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56276b2e-8546-488f-a878-4e3e071274b5-tls-certs\") pod \"auth-test-model-b73303-kserve-598f4bb9b7-lqsch\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:48.607709 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.607565 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-dshm\") pod \"auth-test-model-b73303-kserve-598f4bb9b7-lqsch\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:48.607709 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.607593 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-model-cache\") pod \"auth-test-model-b73303-kserve-598f4bb9b7-lqsch\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:48.607709 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.607628 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-home\") pod \"auth-test-model-b73303-kserve-598f4bb9b7-lqsch\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:48.607709 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.607652 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-tmp-dir\") pod \"auth-test-model-b73303-kserve-598f4bb9b7-lqsch\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:48.607709 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.607679 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dccgb\" (UniqueName: \"kubernetes.io/projected/56276b2e-8546-488f-a878-4e3e071274b5-kube-api-access-dccgb\") pod \"auth-test-model-b73303-kserve-598f4bb9b7-lqsch\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:48.608091 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.608069 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-model-cache\") pod \"auth-test-model-b73303-kserve-598f4bb9b7-lqsch\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:48.608169 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.608108 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-home\") pod \"auth-test-model-b73303-kserve-598f4bb9b7-lqsch\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:48.608169 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.608124 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-tmp-dir\") pod \"auth-test-model-b73303-kserve-598f4bb9b7-lqsch\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:48.610050 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.610032 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-dshm\") pod \"auth-test-model-b73303-kserve-598f4bb9b7-lqsch\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:48.610205 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.610187 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56276b2e-8546-488f-a878-4e3e071274b5-tls-certs\") pod \"auth-test-model-b73303-kserve-598f4bb9b7-lqsch\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:48.619161 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.619140 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dccgb\" (UniqueName: \"kubernetes.io/projected/56276b2e-8546-488f-a878-4e3e071274b5-kube-api-access-dccgb\") pod \"auth-test-model-b73303-kserve-598f4bb9b7-lqsch\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:48.722517 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.722439 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:48.733484 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:48.733453 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:49.019066 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:49.019041 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch"] Sep 10 16:24:49.021085 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:24:49.021048 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56276b2e_8546_488f_a878_4e3e071274b5.slice/crio-848fa77436e1209a0ee45728d5966288cbecc97f852c0becb4d4c3220f2878a8 WatchSource:0}: Error finding container 848fa77436e1209a0ee45728d5966288cbecc97f852c0becb4d4c3220f2878a8: Status 404 returned error can't find the container with id 848fa77436e1209a0ee45728d5966288cbecc97f852c0becb4d4c3220f2878a8 Sep 10 16:24:49.259599 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:49.259560 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" event={"ID":"56276b2e-8546-488f-a878-4e3e071274b5","Type":"ContainerStarted","Data":"f6d16b6f65f74adc4bb2297f1eaf4ba8331acfb3217b5242c2ca164711d4f1ad"} Sep 10 16:24:49.259599 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:49.259605 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" event={"ID":"56276b2e-8546-488f-a878-4e3e071274b5","Type":"ContainerStarted","Data":"848fa77436e1209a0ee45728d5966288cbecc97f852c0becb4d4c3220f2878a8"} Sep 10 16:24:49.278513 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:49.278344 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" podStartSLOduration=1.27832202 podStartE2EDuration="1.27832202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:24:49.277970128 +0000 UTC m=+1388.801215326" watchObservedRunningTime="2026-09-10 16:24:49.27832202 +0000 UTC m=+1388.801567219" Sep 10 16:24:51.762802 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:51.762760 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:24:58.734249 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:58.734199 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:58.734706 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:58.734269 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:58.747038 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:58.746999 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:24:59.328184 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:24:59.328156 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:25:07.807215 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:07.807179 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:08.170806 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.170770 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:08.488521 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.488447 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4"] Sep 10 16:25:08.492568 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.492546 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.495438 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.495393 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-90bd7e-openshift-default-dockercfg-72x45\"" Sep 10 16:25:08.509796 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.509761 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4"] Sep 10 16:25:08.546882 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.546843 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:08.597533 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.597496 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9qzx\" (UniqueName: \"kubernetes.io/projected/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-kube-api-access-k9qzx\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.597742 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.597551 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-data\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.597742 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.597686 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-workload-certs\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.597874 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.597742 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-envoy\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.597932 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.597871 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-credential-socket\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.597988 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.597948 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-podinfo\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.598119 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.598095 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istiod-ca-cert\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.598201 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.598162 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-workload-socket\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.598255 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.598211 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-token\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.699204 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.699168 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istiod-ca-cert\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.699421 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.699214 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-workload-socket\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.699421 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.699277 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-token\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.699421 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.699343 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9qzx\" (UniqueName: \"kubernetes.io/projected/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-kube-api-access-k9qzx\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.699421 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.699365 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-data\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.699421 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.699388 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-workload-certs\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.699421 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.699420 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-envoy\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.699821 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.699445 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-credential-socket\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.699821 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.699561 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-podinfo\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.699821 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.699720 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-data\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.699821 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.699809 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-workload-certs\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.700008 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.699910 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-credential-socket\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.700008 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.699929 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-workload-socket\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.700008 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.699947 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istiod-ca-cert\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.701836 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.701813 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-envoy\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.702202 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.702184 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-podinfo\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.709102 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.709078 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-token\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.709205 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.709193 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9qzx\" (UniqueName: \"kubernetes.io/projected/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-kube-api-access-k9qzx\") pod \"e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:08.804780 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:08.804682 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:09.091620 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:09.091587 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4"] Sep 10 16:25:09.093289 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:25:09.093261 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeef5b3d_8ba8_4901_a1d8_9135ec7b5092.slice/crio-9a9477147287acd7d530da32d8b9268ec353728d0dafc2550288eb6f64df868c WatchSource:0}: Error finding container 9a9477147287acd7d530da32d8b9268ec353728d0dafc2550288eb6f64df868c: Status 404 returned error can't find the container with id 9a9477147287acd7d530da32d8b9268ec353728d0dafc2550288eb6f64df868c Sep 10 16:25:09.095805 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:09.095766 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:25:09.095915 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:09.095837 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:25:09.095915 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:09.095868 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:25:09.378582 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:09.378543 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" event={"ID":"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092","Type":"ContainerStarted","Data":"5a2ade10d48a325ea24de182bf0f085cb32b2d0ef8f5a29ae07e789a652dae6c"} Sep 10 16:25:09.378774 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:09.378607 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" event={"ID":"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092","Type":"ContainerStarted","Data":"9a9477147287acd7d530da32d8b9268ec353728d0dafc2550288eb6f64df868c"} Sep 10 16:25:09.412789 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:09.412725 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" podStartSLOduration=1.4127017579999999 podStartE2EDuration="1.412701758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:25:09.409813003 +0000 UTC m=+1408.933058223" watchObservedRunningTime="2026-09-10 16:25:09.412701758 +0000 UTC m=+1408.935946956" Sep 10 16:25:09.805679 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:09.805582 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:09.842625 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:09.842591 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:10.810664 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:10.810636 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:11.394473 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:11.394441 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:11.395466 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:11.395442 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:11.472092 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:11.472057 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:11.559967 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:11.559935 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:12.649115 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:12.649083 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:12.749824 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:12.749785 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:12.927191 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:12.927102 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:13.185756 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:13.185661 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:13.284542 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:13.284503 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:13.376280 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:13.376246 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g"] Sep 10 16:25:13.379909 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:13.379887 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g" Sep 10 16:25:13.382822 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:13.382800 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-90bd7e\"" Sep 10 16:25:13.385579 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:13.385556 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-90bd7e-dockercfg-vbjsx\"" Sep 10 16:25:13.412663 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:13.412639 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g"] Sep 10 16:25:13.449755 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:13.449668 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fbc7294-dcdd-4246-8142-0db96e7f6ab8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g\" (UID: \"5fbc7294-dcdd-4246-8142-0db96e7f6ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g" Sep 10 16:25:13.450079 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:13.450048 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6x76\" (UniqueName: \"kubernetes.io/projected/5fbc7294-dcdd-4246-8142-0db96e7f6ab8-kube-api-access-d6x76\") pod \"payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g\" (UID: \"5fbc7294-dcdd-4246-8142-0db96e7f6ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g" Sep 10 16:25:13.551511 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:13.551477 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fbc7294-dcdd-4246-8142-0db96e7f6ab8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g\" (UID: \"5fbc7294-dcdd-4246-8142-0db96e7f6ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g" Sep 10 16:25:13.551692 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:13.551528 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6x76\" (UniqueName: \"kubernetes.io/projected/5fbc7294-dcdd-4246-8142-0db96e7f6ab8-kube-api-access-d6x76\") pod \"payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g\" (UID: \"5fbc7294-dcdd-4246-8142-0db96e7f6ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g" Sep 10 16:25:13.552260 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:13.552237 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fbc7294-dcdd-4246-8142-0db96e7f6ab8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g\" (UID: \"5fbc7294-dcdd-4246-8142-0db96e7f6ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g" Sep 10 16:25:13.562216 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:13.562188 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6x76\" (UniqueName: \"kubernetes.io/projected/5fbc7294-dcdd-4246-8142-0db96e7f6ab8-kube-api-access-d6x76\") pod \"payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g\" (UID: \"5fbc7294-dcdd-4246-8142-0db96e7f6ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g" Sep 10 16:25:13.691824 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:13.691786 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g" Sep 10 16:25:13.986278 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:25:13.986244 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fbc7294_dcdd_4246_8142_0db96e7f6ab8.slice/crio-ca7000514671a755cd5eada1437e5df66d115d64582582f7aa833848eac055e6 WatchSource:0}: Error finding container ca7000514671a755cd5eada1437e5df66d115d64582582f7aa833848eac055e6: Status 404 returned error can't find the container with id ca7000514671a755cd5eada1437e5df66d115d64582582f7aa833848eac055e6 Sep 10 16:25:13.986527 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:13.986501 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g"] Sep 10 16:25:14.415190 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:14.415148 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g" event={"ID":"5fbc7294-dcdd-4246-8142-0db96e7f6ab8","Type":"ContainerStarted","Data":"7601f9c5f7e1fba41c48e568c18576eb6a16731d1103424d63fe83e29a2ca250"} Sep 10 16:25:14.415386 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:14.415214 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g" event={"ID":"5fbc7294-dcdd-4246-8142-0db96e7f6ab8","Type":"ContainerStarted","Data":"ca7000514671a755cd5eada1437e5df66d115d64582582f7aa833848eac055e6"} Sep 10 16:25:14.415386 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:14.415244 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g" Sep 10 16:25:14.440055 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:14.439999 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g" podStartSLOduration=1.439980287 podStartE2EDuration="1.439980287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:25:14.439845625 +0000 UTC m=+1413.963090822" watchObservedRunningTime="2026-09-10 16:25:14.439980287 +0000 UTC m=+1413.963225484" Sep 10 16:25:14.467950 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:14.467913 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:25:25.423887 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:25.423858 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g" Sep 10 16:25:30.953856 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:30.953817 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:25:32.536494 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:32.536412 2541 generic.go:356] "Generic (PLEG): container finished" podID="aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" containerID="5a2ade10d48a325ea24de182bf0f085cb32b2d0ef8f5a29ae07e789a652dae6c" exitCode=0 Sep 10 16:25:32.536907 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:32.536487 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" event={"ID":"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092","Type":"ContainerDied","Data":"5a2ade10d48a325ea24de182bf0f085cb32b2d0ef8f5a29ae07e789a652dae6c"} Sep 10 16:25:32.536907 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:32.536902 2541 scope.go:117] "RemoveContainer" containerID="5a2ade10d48a325ea24de182bf0f085cb32b2d0ef8f5a29ae07e789a652dae6c" Sep 10 16:25:32.538150 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:32.538117 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:25:32.538214 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:32.538195 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:25:32.538253 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:32.538232 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:25:33.544249 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:33.544213 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" event={"ID":"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092","Type":"ContainerStarted","Data":"3c83fe730036df781436136721e2dcd8f08c540e4470d255eeedd5f300f2d221"} Sep 10 16:25:33.806145 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:33.806037 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:33.806145 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:33.806078 2541 patch_prober.go:28] interesting pod/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 10 16:25:33.806145 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:33.806123 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" podUID="aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 10 16:25:34.806124 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:34.806072 2541 patch_prober.go:28] interesting pod/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 10 16:25:34.806660 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:34.806144 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" podUID="aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 10 16:25:35.806028 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:35.805991 2541 patch_prober.go:28] interesting pod/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 10 16:25:35.806260 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:35.806056 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" podUID="aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 10 16:25:36.350835 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:36.350798 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:36.805412 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:36.805370 2541 patch_prober.go:28] interesting pod/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 10 16:25:36.805636 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:36.805444 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" podUID="aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 10 16:25:37.805934 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:37.805895 2541 patch_prober.go:28] interesting pod/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 10 16:25:37.806390 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:37.805970 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" podUID="aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 10 16:25:38.995105 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:38.995073 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:38.995597 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:38.995413 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:38.996088 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:38.996067 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:25:47.637680 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:47.637645 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:47.938520 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:47.938432 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j"] Sep 10 16:25:47.942284 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:47.942263 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:47.945885 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:47.945855 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7dabb3eb-openshift-default-dockercfg-cspkf\"" Sep 10 16:25:47.962249 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:47.962223 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j"] Sep 10 16:25:47.982371 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:47.982344 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:47.997672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:47.997640 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-data\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:47.997931 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:47.997909 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istiod-ca-cert\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:47.998076 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:47.998062 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-token\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:47.998209 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:47.998195 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-credential-socket\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:47.998339 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:47.998324 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-podinfo\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:47.998480 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:47.998465 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-workload-socket\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:47.998580 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:47.998567 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-envoy\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:47.998699 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:47.998687 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-workload-certs\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:47.998825 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:47.998812 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c252j\" (UniqueName: \"kubernetes.io/projected/8cf19c97-1191-4575-9aaa-4a01d89fbb28-kube-api-access-c252j\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:48.100390 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.100351 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-workload-certs\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:48.100600 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.100423 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c252j\" (UniqueName: \"kubernetes.io/projected/8cf19c97-1191-4575-9aaa-4a01d89fbb28-kube-api-access-c252j\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:48.100600 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.100467 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-data\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:48.100600 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.100483 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istiod-ca-cert\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:48.100600 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.100526 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-token\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:48.100600 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.100566 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-credential-socket\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:48.100600 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.100592 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-podinfo\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:48.100924 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.100641 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-workload-socket\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:48.100924 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.100665 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-envoy\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:48.100924 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.100851 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-workload-certs\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:48.101216 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.101185 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-credential-socket\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:48.101531 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.101508 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-data\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:48.101637 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.101508 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-workload-socket\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:48.101713 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.101692 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istiod-ca-cert\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:48.103372 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.103342 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-envoy\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:48.103846 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.103825 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-podinfo\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:48.113733 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.113706 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c252j\" (UniqueName: \"kubernetes.io/projected/8cf19c97-1191-4575-9aaa-4a01d89fbb28-kube-api-access-c252j\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:48.114927 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.114911 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-token\") pod \"e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:48.195667 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.195580 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:48.253707 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.253669 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:48.542944 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.542918 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j"] Sep 10 16:25:48.544488 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:25:48.544460 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf19c97_1191_4575_9aaa_4a01d89fbb28.slice/crio-30214ea9cf7face962577a176b0ab98ef02e94440442e6249bab11618552fc76 WatchSource:0}: Error finding container 30214ea9cf7face962577a176b0ab98ef02e94440442e6249bab11618552fc76: Status 404 returned error can't find the container with id 30214ea9cf7face962577a176b0ab98ef02e94440442e6249bab11618552fc76 Sep 10 16:25:48.546888 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.546852 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:25:48.546989 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.546954 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:25:48.547035 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.546987 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:25:48.641123 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.641088 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" event={"ID":"8cf19c97-1191-4575-9aaa-4a01d89fbb28","Type":"ContainerStarted","Data":"b08ecd25148367d34c7a64e877f307c1c17d41789f017cfabec943b9959acadd"} Sep 10 16:25:48.641607 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.641149 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" event={"ID":"8cf19c97-1191-4575-9aaa-4a01d89fbb28","Type":"ContainerStarted","Data":"30214ea9cf7face962577a176b0ab98ef02e94440442e6249bab11618552fc76"} Sep 10 16:25:48.667757 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.667694 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" podStartSLOduration=1.667654458 podStartE2EDuration="1.667654458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:25:48.666662979 +0000 UTC m=+1448.189908177" watchObservedRunningTime="2026-09-10 16:25:48.667654458 +0000 UTC m=+1448.190899654" Sep 10 16:25:48.879465 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:48.879428 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:49.186578 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:49.186476 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:25:49.254492 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:49.254451 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:49.260209 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:49.260177 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:49.647137 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:49.647109 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:49.648262 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:49.648239 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:25:50.699376 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:50.699342 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:50.813716 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:50.813680 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:51.972420 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:51.972375 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:52.049530 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:52.049495 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:52.274042 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:52.273952 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:52.637538 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:52.637511 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:52.896063 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:25:52.895981 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:26:08.768708 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:08.768674 2541 generic.go:356] "Generic (PLEG): container finished" podID="8cf19c97-1191-4575-9aaa-4a01d89fbb28" containerID="b08ecd25148367d34c7a64e877f307c1c17d41789f017cfabec943b9959acadd" exitCode=0 Sep 10 16:26:08.769128 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:08.768751 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" event={"ID":"8cf19c97-1191-4575-9aaa-4a01d89fbb28","Type":"ContainerDied","Data":"b08ecd25148367d34c7a64e877f307c1c17d41789f017cfabec943b9959acadd"} Sep 10 16:26:08.769190 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:08.769177 2541 scope.go:117] "RemoveContainer" containerID="b08ecd25148367d34c7a64e877f307c1c17d41789f017cfabec943b9959acadd" Sep 10 16:26:08.770507 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:08.770473 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:26:08.770607 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:08.770543 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:26:08.770607 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:08.770581 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:26:09.777475 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:09.777443 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" event={"ID":"8cf19c97-1191-4575-9aaa-4a01d89fbb28","Type":"ContainerStarted","Data":"3d667d257d4abcc8824078bd3864e2a2eca89adf855fb8b1050784e164c54557"} Sep 10 16:26:10.253960 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:10.253924 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:26:10.254252 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:10.254226 2541 patch_prober.go:28] interesting pod/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 10 16:26:10.254378 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:10.254286 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" podUID="8cf19c97-1191-4575-9aaa-4a01d89fbb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 10 16:26:11.254796 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:11.254756 2541 patch_prober.go:28] interesting pod/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 10 16:26:11.255199 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:11.254820 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" podUID="8cf19c97-1191-4575-9aaa-4a01d89fbb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 10 16:26:12.254197 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:12.254156 2541 patch_prober.go:28] interesting pod/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 10 16:26:12.254411 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:12.254214 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" podUID="8cf19c97-1191-4575-9aaa-4a01d89fbb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 10 16:26:12.871749 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:12.871697 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:26:13.254597 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:13.254513 2541 patch_prober.go:28] interesting pod/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 10 16:26:13.254597 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:13.254567 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" podUID="8cf19c97-1191-4575-9aaa-4a01d89fbb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 10 16:26:14.254664 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:14.254618 2541 patch_prober.go:28] interesting pod/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 10 16:26:14.255078 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:14.254685 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" podUID="8cf19c97-1191-4575-9aaa-4a01d89fbb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 10 16:26:15.258532 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:15.258501 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:26:15.258915 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:15.258777 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:26:15.259559 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:15.259543 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:26:15.645912 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:15.645876 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:26:26.150454 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.150418 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:26:26.415436 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.415354 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2"] Sep 10 16:26:26.419420 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.419402 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:26.421694 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.421670 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7dabb3eb\"/\"openshift-service-ca.crt\"" Sep 10 16:26:26.421843 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.421787 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7dabb3eb\"/\"default-dockercfg-86swf\"" Sep 10 16:26:26.422481 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.422423 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7dabb3eb\"/\"e2e-lifecycle-model-7dabb3eb-kserve-self-signed-certs\"" Sep 10 16:26:26.422582 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.422484 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7dabb3eb\"/\"kube-root-ca.crt\"" Sep 10 16:26:26.430838 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.430815 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2"] Sep 10 16:26:26.446197 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.446170 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:26:26.585802 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.585764 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-model-cache\") pod \"e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:26.586165 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.586128 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-tmp-dir\") pod \"e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:26.586283 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.586224 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-tls-certs\") pod \"e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:26.586283 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.586267 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-dshm\") pod \"e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:26.586425 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.586346 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-home\") pod \"e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:26.586509 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.586480 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrrqw\" (UniqueName: \"kubernetes.io/projected/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-kube-api-access-wrrqw\") pod \"e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:26.687342 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.687200 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-home\") pod \"e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:26.687342 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.687269 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrrqw\" (UniqueName: \"kubernetes.io/projected/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-kube-api-access-wrrqw\") pod \"e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:26.687604 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.687344 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-model-cache\") pod \"e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:26.687604 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.687404 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-tmp-dir\") pod \"e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:26.687604 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.687437 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-tls-certs\") pod \"e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:26.687604 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.687456 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-dshm\") pod \"e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:26.687819 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.687629 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-home\") pod \"e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:26.687819 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.687677 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-model-cache\") pod \"e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:26.687819 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.687746 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-tmp-dir\") pod \"e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:26.689888 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.689860 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-dshm\") pod \"e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:26.690247 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.690224 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-tls-certs\") pod \"e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:26.697273 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.697250 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrrqw\" (UniqueName: \"kubernetes.io/projected/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-kube-api-access-wrrqw\") pod \"e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:26.730378 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:26.730349 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:27.014989 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:27.014959 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2"] Sep 10 16:26:27.017016 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:26:27.016988 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b05adb8_b2cb_47b1_b0e1_afc8b86abcf5.slice/crio-d7eb3f7b2adef19c09d07052bff69868150919702d9e99c0e3e8e172b170a20a WatchSource:0}: Error finding container d7eb3f7b2adef19c09d07052bff69868150919702d9e99c0e3e8e172b170a20a: Status 404 returned error can't find the container with id d7eb3f7b2adef19c09d07052bff69868150919702d9e99c0e3e8e172b170a20a Sep 10 16:26:27.909081 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:27.909043 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" event={"ID":"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5","Type":"ContainerStarted","Data":"58b6469172311e5db1072c9a018e0a6623ac136127925adebda7304fda757237"} Sep 10 16:26:27.909081 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:27.909083 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" event={"ID":"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5","Type":"ContainerStarted","Data":"d7eb3f7b2adef19c09d07052bff69868150919702d9e99c0e3e8e172b170a20a"} Sep 10 16:26:27.928536 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:27.928483 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" podStartSLOduration=1.928465318 podStartE2EDuration="1.928465318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:26:27.92783388 +0000 UTC m=+1487.451079078" watchObservedRunningTime="2026-09-10 16:26:27.928465318 +0000 UTC m=+1487.451710516" Sep 10 16:26:30.863141 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:30.863106 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:26:36.730763 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:36.730725 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:36.731170 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:36.730788 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:36.743475 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:36.743451 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:36.980102 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:36.980070 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:26:41.086894 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:41.086865 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/1.log" Sep 10 16:26:41.090163 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:41.090140 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/1.log" Sep 10 16:26:41.093490 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:41.093473 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4c6j_2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd/ovn-acl-logging/0.log" Sep 10 16:26:41.096601 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:41.096582 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4c6j_2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd/ovn-acl-logging/0.log" Sep 10 16:26:48.053457 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:48.053414 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:26:55.288896 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:55.288858 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:26:58.628106 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:26:58.628054 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:27:06.554239 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:06.554159 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:27:10.419450 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:10.419405 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:27:10.965639 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:10.965597 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:27:11.112495 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:11.112459 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:27:11.210231 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:11.210194 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:27:11.336703 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:11.336619 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:27:11.436288 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:11.436250 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:27:12.796329 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:12.796267 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:27:13.961091 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:13.961051 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch"] Sep 10 16:27:13.974454 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:13.974425 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:27:14.528659 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:14.528625 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:27:19.487674 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:19.487640 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2"] Sep 10 16:27:19.601737 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:19.601695 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:27:25.002142 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:25.002101 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:27:28.272643 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:28.272595 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:27:28.984090 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:28.984021 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" podUID="56276b2e-8546-488f-a878-4e3e071274b5" containerName="main" containerID="cri-o://f6d16b6f65f74adc4bb2297f1eaf4ba8331acfb3217b5242c2ca164711d4f1ad" gracePeriod=45 Sep 10 16:27:29.243898 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.243843 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:27:29.295977 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.295944 2541 generic.go:356] "Generic (PLEG): container finished" podID="56276b2e-8546-488f-a878-4e3e071274b5" containerID="f6d16b6f65f74adc4bb2297f1eaf4ba8331acfb3217b5242c2ca164711d4f1ad" exitCode=0 Sep 10 16:27:29.296398 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.295996 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" Sep 10 16:27:29.296398 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.296037 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" event={"ID":"56276b2e-8546-488f-a878-4e3e071274b5","Type":"ContainerDied","Data":"f6d16b6f65f74adc4bb2297f1eaf4ba8331acfb3217b5242c2ca164711d4f1ad"} Sep 10 16:27:29.296398 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.296088 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch" event={"ID":"56276b2e-8546-488f-a878-4e3e071274b5","Type":"ContainerDied","Data":"848fa77436e1209a0ee45728d5966288cbecc97f852c0becb4d4c3220f2878a8"} Sep 10 16:27:29.296398 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.296109 2541 scope.go:117] "RemoveContainer" containerID="f6d16b6f65f74adc4bb2297f1eaf4ba8331acfb3217b5242c2ca164711d4f1ad" Sep 10 16:27:29.307287 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.307264 2541 scope.go:117] "RemoveContainer" containerID="f6d16b6f65f74adc4bb2297f1eaf4ba8331acfb3217b5242c2ca164711d4f1ad" Sep 10 16:27:29.307621 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:27:29.307603 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d16b6f65f74adc4bb2297f1eaf4ba8331acfb3217b5242c2ca164711d4f1ad\": container with ID starting with f6d16b6f65f74adc4bb2297f1eaf4ba8331acfb3217b5242c2ca164711d4f1ad not found: ID does not exist" containerID="f6d16b6f65f74adc4bb2297f1eaf4ba8331acfb3217b5242c2ca164711d4f1ad" Sep 10 16:27:29.307712 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.307631 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d16b6f65f74adc4bb2297f1eaf4ba8331acfb3217b5242c2ca164711d4f1ad"} err="failed to get container status \"f6d16b6f65f74adc4bb2297f1eaf4ba8331acfb3217b5242c2ca164711d4f1ad\": rpc error: code = NotFound desc = could not find container \"f6d16b6f65f74adc4bb2297f1eaf4ba8331acfb3217b5242c2ca164711d4f1ad\": container with ID starting with f6d16b6f65f74adc4bb2297f1eaf4ba8331acfb3217b5242c2ca164711d4f1ad not found: ID does not exist" Sep 10 16:27:29.314405 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.314382 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56276b2e-8546-488f-a878-4e3e071274b5-tls-certs\") pod \"56276b2e-8546-488f-a878-4e3e071274b5\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " Sep 10 16:27:29.314500 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.314423 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-home\") pod \"56276b2e-8546-488f-a878-4e3e071274b5\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " Sep 10 16:27:29.314558 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.314545 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-tmp-dir\") pod \"56276b2e-8546-488f-a878-4e3e071274b5\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " Sep 10 16:27:29.314622 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.314573 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-model-cache\") pod \"56276b2e-8546-488f-a878-4e3e071274b5\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " Sep 10 16:27:29.314622 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.314591 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-dshm\") pod \"56276b2e-8546-488f-a878-4e3e071274b5\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " Sep 10 16:27:29.314727 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.314619 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dccgb\" (UniqueName: \"kubernetes.io/projected/56276b2e-8546-488f-a878-4e3e071274b5-kube-api-access-dccgb\") pod \"56276b2e-8546-488f-a878-4e3e071274b5\" (UID: \"56276b2e-8546-488f-a878-4e3e071274b5\") " Sep 10 16:27:29.314785 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.314762 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-home" (OuterVolumeSpecName: "home") pod "56276b2e-8546-488f-a878-4e3e071274b5" (UID: "56276b2e-8546-488f-a878-4e3e071274b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:27:29.314839 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.314816 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56276b2e-8546-488f-a878-4e3e071274b5" (UID: "56276b2e-8546-488f-a878-4e3e071274b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:27:29.314839 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.314826 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-model-cache" (OuterVolumeSpecName: "model-cache") pod "56276b2e-8546-488f-a878-4e3e071274b5" (UID: "56276b2e-8546-488f-a878-4e3e071274b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:27:29.315012 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.314993 2541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-home\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:27:29.315247 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.315181 2541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-tmp-dir\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:27:29.315247 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.315225 2541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-model-cache\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:27:29.316876 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.316851 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-dshm" (OuterVolumeSpecName: "dshm") pod "56276b2e-8546-488f-a878-4e3e071274b5" (UID: "56276b2e-8546-488f-a878-4e3e071274b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:27:29.316876 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.316861 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56276b2e-8546-488f-a878-4e3e071274b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56276b2e-8546-488f-a878-4e3e071274b5" (UID: "56276b2e-8546-488f-a878-4e3e071274b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:27:29.317002 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.316946 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56276b2e-8546-488f-a878-4e3e071274b5-kube-api-access-dccgb" (OuterVolumeSpecName: "kube-api-access-dccgb") pod "56276b2e-8546-488f-a878-4e3e071274b5" (UID: "56276b2e-8546-488f-a878-4e3e071274b5"). InnerVolumeSpecName "kube-api-access-dccgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:27:29.416653 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.416609 2541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56276b2e-8546-488f-a878-4e3e071274b5-dshm\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:27:29.416653 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.416651 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dccgb\" (UniqueName: \"kubernetes.io/projected/56276b2e-8546-488f-a878-4e3e071274b5-kube-api-access-dccgb\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:27:29.416653 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.416662 2541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56276b2e-8546-488f-a878-4e3e071274b5-tls-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:27:29.636408 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.636376 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch"] Sep 10 16:27:29.648153 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:29.648125 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b73303/auth-test-model-b73303-kserve-598f4bb9b7-lqsch"] Sep 10 16:27:30.981935 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:30.981890 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56276b2e-8546-488f-a878-4e3e071274b5" path="/var/lib/kubelet/pods/56276b2e-8546-488f-a878-4e3e071274b5/volumes" Sep 10 16:27:34.509082 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:34.509033 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" podUID="4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5" containerName="main" containerID="cri-o://58b6469172311e5db1072c9a018e0a6623ac136127925adebda7304fda757237" gracePeriod=45 Sep 10 16:27:34.770893 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:34.770817 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:27:34.867684 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:34.867648 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-tmp-dir\") pod \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " Sep 10 16:27:34.867870 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:34.867716 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-home\") pod \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " Sep 10 16:27:34.867870 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:34.867775 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-tls-certs\") pod \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " Sep 10 16:27:34.867870 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:34.867799 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrrqw\" (UniqueName: \"kubernetes.io/projected/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-kube-api-access-wrrqw\") pod \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " Sep 10 16:27:34.868020 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:34.867949 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-dshm\") pod \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " Sep 10 16:27:34.868020 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:34.867995 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5" (UID: "4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:27:34.868020 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:34.868004 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-home" (OuterVolumeSpecName: "home") pod "4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5" (UID: "4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:27:34.868178 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:34.868019 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-model-cache\") pod \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\" (UID: \"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5\") " Sep 10 16:27:34.868244 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:34.868217 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5" (UID: "4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:27:34.868552 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:34.868434 2541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-home\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:27:34.868552 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:34.868455 2541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-model-cache\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:27:34.868552 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:34.868464 2541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-tmp-dir\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:27:34.870543 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:34.870513 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-dshm" (OuterVolumeSpecName: "dshm") pod "4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5" (UID: "4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:27:34.870747 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:34.870719 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5" (UID: "4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:27:34.870827 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:34.870725 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-kube-api-access-wrrqw" (OuterVolumeSpecName: "kube-api-access-wrrqw") pod "4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5" (UID: "4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5"). InnerVolumeSpecName "kube-api-access-wrrqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:27:34.969148 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:34.969106 2541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-tls-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:27:34.969148 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:34.969140 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrrqw\" (UniqueName: \"kubernetes.io/projected/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-kube-api-access-wrrqw\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:27:34.969148 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:34.969150 2541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5-dshm\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:27:35.335691 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:35.335654 2541 generic.go:356] "Generic (PLEG): container finished" podID="4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5" containerID="58b6469172311e5db1072c9a018e0a6623ac136127925adebda7304fda757237" exitCode=0 Sep 10 16:27:35.335877 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:35.335703 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" event={"ID":"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5","Type":"ContainerDied","Data":"58b6469172311e5db1072c9a018e0a6623ac136127925adebda7304fda757237"} Sep 10 16:27:35.335877 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:35.335714 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" Sep 10 16:27:35.335877 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:35.335735 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2" event={"ID":"4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5","Type":"ContainerDied","Data":"d7eb3f7b2adef19c09d07052bff69868150919702d9e99c0e3e8e172b170a20a"} Sep 10 16:27:35.335877 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:35.335753 2541 scope.go:117] "RemoveContainer" containerID="58b6469172311e5db1072c9a018e0a6623ac136127925adebda7304fda757237" Sep 10 16:27:35.345105 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:35.345081 2541 scope.go:117] "RemoveContainer" containerID="58b6469172311e5db1072c9a018e0a6623ac136127925adebda7304fda757237" Sep 10 16:27:35.345435 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:27:35.345409 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b6469172311e5db1072c9a018e0a6623ac136127925adebda7304fda757237\": container with ID starting with 58b6469172311e5db1072c9a018e0a6623ac136127925adebda7304fda757237 not found: ID does not exist" containerID="58b6469172311e5db1072c9a018e0a6623ac136127925adebda7304fda757237" Sep 10 16:27:35.345519 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:35.345447 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b6469172311e5db1072c9a018e0a6623ac136127925adebda7304fda757237"} err="failed to get container status \"58b6469172311e5db1072c9a018e0a6623ac136127925adebda7304fda757237\": rpc error: code = NotFound desc = could not find container \"58b6469172311e5db1072c9a018e0a6623ac136127925adebda7304fda757237\": container with ID starting with 58b6469172311e5db1072c9a018e0a6623ac136127925adebda7304fda757237 not found: ID does not exist" Sep 10 16:27:35.354978 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:35.354947 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2"] Sep 10 16:27:35.362877 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:35.362853 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7dabb3eb/e2e-lifecycle-model-7dabb3eb-kserve-d7594cf95-xnpw2"] Sep 10 16:27:36.980891 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:36.980855 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5" path="/var/lib/kubelet/pods/4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5/volumes" Sep 10 16:27:41.201866 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:41.201836 2541 scope.go:117] "RemoveContainer" containerID="0343f6f84e34d5c91938222bce992621c034165ba62e2701299a6bda0e350f91" Sep 10 16:27:41.211125 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:41.211105 2541 scope.go:117] "RemoveContainer" containerID="1a4f589e2117724acb0787fd540c2a2e969680b8592c93fc133ecb2b92b6f2bd" Sep 10 16:27:47.815241 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:47.815199 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv"] Sep 10 16:27:47.815886 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:47.815864 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5" containerName="main" Sep 10 16:27:47.815950 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:47.815889 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5" containerName="main" Sep 10 16:27:47.815950 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:47.815905 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56276b2e-8546-488f-a878-4e3e071274b5" containerName="main" Sep 10 16:27:47.815950 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:47.815910 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="56276b2e-8546-488f-a878-4e3e071274b5" containerName="main" Sep 10 16:27:47.816049 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:47.815974 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b05adb8-b2cb-47b1-b0e1-afc8b86abcf5" containerName="main" Sep 10 16:27:47.816049 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:47.815985 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="56276b2e-8546-488f-a878-4e3e071274b5" containerName="main" Sep 10 16:27:47.820816 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:47.820795 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:47.823183 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:47.823159 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b73303\"/\"openshift-service-ca.crt\"" Sep 10 16:27:47.823293 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:47.823216 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b73303\"/\"kube-root-ca.crt\"" Sep 10 16:27:47.823800 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:47.823781 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b73303\"/\"default-dockercfg-2jzpz\"" Sep 10 16:27:47.823908 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:47.823823 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b73303\"/\"sub-test-model-b73303-kserve-self-signed-certs\"" Sep 10 16:27:47.830286 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:47.830261 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv"] Sep 10 16:27:47.853180 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:47.853148 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:27:47.999431 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:47.999386 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-home\") pod \"sub-test-model-b73303-kserve-db4549667-qchkv\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:47.999431 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:47.999433 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-model-cache\") pod \"sub-test-model-b73303-kserve-db4549667-qchkv\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:47.999709 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:47.999464 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-dshm\") pod \"sub-test-model-b73303-kserve-db4549667-qchkv\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:47.999709 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:47.999569 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-tmp-dir\") pod \"sub-test-model-b73303-kserve-db4549667-qchkv\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:47.999709 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:47.999604 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-tls-certs\") pod \"sub-test-model-b73303-kserve-db4549667-qchkv\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:47.999869 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:47.999723 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp74z\" (UniqueName: \"kubernetes.io/projected/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-kube-api-access-dp74z\") pod \"sub-test-model-b73303-kserve-db4549667-qchkv\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:48.101077 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:48.100974 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp74z\" (UniqueName: \"kubernetes.io/projected/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-kube-api-access-dp74z\") pod \"sub-test-model-b73303-kserve-db4549667-qchkv\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:48.101077 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:48.101073 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-home\") pod \"sub-test-model-b73303-kserve-db4549667-qchkv\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:48.101331 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:48.101099 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-model-cache\") pod \"sub-test-model-b73303-kserve-db4549667-qchkv\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:48.101331 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:48.101122 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-dshm\") pod \"sub-test-model-b73303-kserve-db4549667-qchkv\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:48.101331 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:48.101180 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-tmp-dir\") pod \"sub-test-model-b73303-kserve-db4549667-qchkv\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:48.101331 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:48.101201 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-tls-certs\") pod \"sub-test-model-b73303-kserve-db4549667-qchkv\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:48.101606 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:48.101564 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-home\") pod \"sub-test-model-b73303-kserve-db4549667-qchkv\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:48.101606 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:48.101590 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-model-cache\") pod \"sub-test-model-b73303-kserve-db4549667-qchkv\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:48.101705 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:48.101623 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-tmp-dir\") pod \"sub-test-model-b73303-kserve-db4549667-qchkv\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:48.103702 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:48.103681 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-dshm\") pod \"sub-test-model-b73303-kserve-db4549667-qchkv\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:48.103830 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:48.103813 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-tls-certs\") pod \"sub-test-model-b73303-kserve-db4549667-qchkv\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:48.111381 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:48.111359 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp74z\" (UniqueName: \"kubernetes.io/projected/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-kube-api-access-dp74z\") pod \"sub-test-model-b73303-kserve-db4549667-qchkv\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:48.132428 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:48.132403 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:48.412246 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:48.412215 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv"] Sep 10 16:27:48.414475 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:27:48.414448 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76abd234_f8ea_4ebe_bfa9_1a595ad92f94.slice/crio-9098b1c385e2b1ce3617fee4f3c9649c99720459a7458d812fc632d184732bc9 WatchSource:0}: Error finding container 9098b1c385e2b1ce3617fee4f3c9649c99720459a7458d812fc632d184732bc9: Status 404 returned error can't find the container with id 9098b1c385e2b1ce3617fee4f3c9649c99720459a7458d812fc632d184732bc9 Sep 10 16:27:49.421212 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:49.421173 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" event={"ID":"76abd234-f8ea-4ebe-bfa9-1a595ad92f94","Type":"ContainerStarted","Data":"e71d4a246f7ac47cc171ee30d83bae7cedd6775bd3b0a5d9d06b6f2e2372651e"} Sep 10 16:27:49.421659 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:49.421223 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" event={"ID":"76abd234-f8ea-4ebe-bfa9-1a595ad92f94","Type":"ContainerStarted","Data":"9098b1c385e2b1ce3617fee4f3c9649c99720459a7458d812fc632d184732bc9"} Sep 10 16:27:49.442966 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:49.442914 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" podStartSLOduration=2.442896333 podStartE2EDuration="2.442896333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:27:49.44067163 +0000 UTC m=+1568.963916827" watchObservedRunningTime="2026-09-10 16:27:49.442896333 +0000 UTC m=+1568.966141529" Sep 10 16:27:52.479583 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:52.479541 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:27:56.500333 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:56.500276 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:27:57.648378 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:57.648335 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:27:58.132579 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:58.132543 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:58.132764 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:58.132608 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:58.145332 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:58.145277 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:27:58.488561 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:27:58.488486 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:28:03.539781 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:03.539735 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:03.648492 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:03.648451 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:03.806743 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:03.806662 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j"] Sep 10 16:28:03.806901 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:03.806874 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" podUID="8cf19c97-1191-4575-9aaa-4a01d89fbb28" containerName="istio-proxy" containerID="cri-o://3d667d257d4abcc8824078bd3864e2a2eca89adf855fb8b1050784e164c54557" gracePeriod=30 Sep 10 16:28:03.829351 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:03.829317 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:05.139994 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:05.139956 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:05.239831 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:05.239801 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:05.341388 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:05.341347 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:06.001688 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:06.001655 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:06.120995 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:06.120960 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:07.514569 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:07.514533 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:09.060454 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.060430 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:28:09.209635 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.209598 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-data\") pod \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " Sep 10 16:28:09.209635 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.209636 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-credential-socket\") pod \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " Sep 10 16:28:09.209885 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.209652 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-envoy\") pod \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " Sep 10 16:28:09.209885 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.209706 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-podinfo\") pod \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " Sep 10 16:28:09.209885 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.209723 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istiod-ca-cert\") pod \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " Sep 10 16:28:09.209885 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.209742 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-workload-certs\") pod \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " Sep 10 16:28:09.210058 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.209893 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-workload-socket\") pod \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " Sep 10 16:28:09.210058 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.209919 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-token\") pod \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " Sep 10 16:28:09.210058 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.209973 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8cf19c97-1191-4575-9aaa-4a01d89fbb28" (UID: "8cf19c97-1191-4575-9aaa-4a01d89fbb28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:28:09.210058 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.209998 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-data" (OuterVolumeSpecName: "istio-data") pod "8cf19c97-1191-4575-9aaa-4a01d89fbb28" (UID: "8cf19c97-1191-4575-9aaa-4a01d89fbb28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:28:09.210058 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.209986 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c252j\" (UniqueName: \"kubernetes.io/projected/8cf19c97-1191-4575-9aaa-4a01d89fbb28-kube-api-access-c252j\") pod \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\" (UID: \"8cf19c97-1191-4575-9aaa-4a01d89fbb28\") " Sep 10 16:28:09.210252 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.210137 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8cf19c97-1191-4575-9aaa-4a01d89fbb28" (UID: "8cf19c97-1191-4575-9aaa-4a01d89fbb28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:28:09.210318 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.210254 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8cf19c97-1191-4575-9aaa-4a01d89fbb28" (UID: "8cf19c97-1191-4575-9aaa-4a01d89fbb28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:28:09.210445 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.210429 2541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-workload-socket\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:28:09.210554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.210526 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8cf19c97-1191-4575-9aaa-4a01d89fbb28" (UID: "8cf19c97-1191-4575-9aaa-4a01d89fbb28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:28:09.210554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.210535 2541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-data\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:28:09.210719 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.210579 2541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-credential-socket\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:28:09.210719 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.210596 2541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istiod-ca-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:28:09.212215 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.212195 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-token" (OuterVolumeSpecName: "istio-token") pod "8cf19c97-1191-4575-9aaa-4a01d89fbb28" (UID: "8cf19c97-1191-4575-9aaa-4a01d89fbb28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:28:09.212437 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.212414 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8cf19c97-1191-4575-9aaa-4a01d89fbb28" (UID: "8cf19c97-1191-4575-9aaa-4a01d89fbb28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:28:09.212729 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.212697 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf19c97-1191-4575-9aaa-4a01d89fbb28-kube-api-access-c252j" (OuterVolumeSpecName: "kube-api-access-c252j") pod "8cf19c97-1191-4575-9aaa-4a01d89fbb28" (UID: "8cf19c97-1191-4575-9aaa-4a01d89fbb28"). InnerVolumeSpecName "kube-api-access-c252j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:28:09.212816 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.212721 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8cf19c97-1191-4575-9aaa-4a01d89fbb28" (UID: "8cf19c97-1191-4575-9aaa-4a01d89fbb28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:28:09.311412 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.311372 2541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-token\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:28:09.311412 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.311404 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c252j\" (UniqueName: \"kubernetes.io/projected/8cf19c97-1191-4575-9aaa-4a01d89fbb28-kube-api-access-c252j\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:28:09.311412 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.311414 2541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-envoy\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:28:09.311412 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.311427 2541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf19c97-1191-4575-9aaa-4a01d89fbb28-istio-podinfo\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:28:09.311705 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.311437 2541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf19c97-1191-4575-9aaa-4a01d89fbb28-workload-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:28:09.546505 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.546416 2541 generic.go:356] "Generic (PLEG): container finished" podID="8cf19c97-1191-4575-9aaa-4a01d89fbb28" containerID="3d667d257d4abcc8824078bd3864e2a2eca89adf855fb8b1050784e164c54557" exitCode=0 Sep 10 16:28:09.546505 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.546468 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" Sep 10 16:28:09.546505 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.546484 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" event={"ID":"8cf19c97-1191-4575-9aaa-4a01d89fbb28","Type":"ContainerDied","Data":"3d667d257d4abcc8824078bd3864e2a2eca89adf855fb8b1050784e164c54557"} Sep 10 16:28:09.546778 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.546540 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j" event={"ID":"8cf19c97-1191-4575-9aaa-4a01d89fbb28","Type":"ContainerDied","Data":"30214ea9cf7face962577a176b0ab98ef02e94440442e6249bab11618552fc76"} Sep 10 16:28:09.546778 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.546567 2541 scope.go:117] "RemoveContainer" containerID="3d667d257d4abcc8824078bd3864e2a2eca89adf855fb8b1050784e164c54557" Sep 10 16:28:09.556908 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.556886 2541 scope.go:117] "RemoveContainer" containerID="b08ecd25148367d34c7a64e877f307c1c17d41789f017cfabec943b9959acadd" Sep 10 16:28:09.566348 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.566327 2541 scope.go:117] "RemoveContainer" containerID="3d667d257d4abcc8824078bd3864e2a2eca89adf855fb8b1050784e164c54557" Sep 10 16:28:09.566706 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:28:09.566686 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d667d257d4abcc8824078bd3864e2a2eca89adf855fb8b1050784e164c54557\": container with ID starting with 3d667d257d4abcc8824078bd3864e2a2eca89adf855fb8b1050784e164c54557 not found: ID does not exist" containerID="3d667d257d4abcc8824078bd3864e2a2eca89adf855fb8b1050784e164c54557" Sep 10 16:28:09.566756 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.566717 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d667d257d4abcc8824078bd3864e2a2eca89adf855fb8b1050784e164c54557"} err="failed to get container status \"3d667d257d4abcc8824078bd3864e2a2eca89adf855fb8b1050784e164c54557\": rpc error: code = NotFound desc = could not find container \"3d667d257d4abcc8824078bd3864e2a2eca89adf855fb8b1050784e164c54557\": container with ID starting with 3d667d257d4abcc8824078bd3864e2a2eca89adf855fb8b1050784e164c54557 not found: ID does not exist" Sep 10 16:28:09.566756 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.566736 2541 scope.go:117] "RemoveContainer" containerID="b08ecd25148367d34c7a64e877f307c1c17d41789f017cfabec943b9959acadd" Sep 10 16:28:09.567007 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:28:09.566988 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b08ecd25148367d34c7a64e877f307c1c17d41789f017cfabec943b9959acadd\": container with ID starting with b08ecd25148367d34c7a64e877f307c1c17d41789f017cfabec943b9959acadd not found: ID does not exist" containerID="b08ecd25148367d34c7a64e877f307c1c17d41789f017cfabec943b9959acadd" Sep 10 16:28:09.567053 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.567015 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08ecd25148367d34c7a64e877f307c1c17d41789f017cfabec943b9959acadd"} err="failed to get container status \"b08ecd25148367d34c7a64e877f307c1c17d41789f017cfabec943b9959acadd\": rpc error: code = NotFound desc = could not find container \"b08ecd25148367d34c7a64e877f307c1c17d41789f017cfabec943b9959acadd\": container with ID starting with b08ecd25148367d34c7a64e877f307c1c17d41789f017cfabec943b9959acadd not found: ID does not exist" Sep 10 16:28:09.575689 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.575661 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j"] Sep 10 16:28:09.579947 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.579915 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7dabb3eb-openshift-default-5d7b99f69b-m695j"] Sep 10 16:28:09.959642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:09.959602 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:28:10.981724 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:10.981688 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cf19c97-1191-4575-9aaa-4a01d89fbb28" path="/var/lib/kubelet/pods/8cf19c97-1191-4575-9aaa-4a01d89fbb28/volumes" Sep 10 16:28:13.973152 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:13.973112 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:14.062942 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:14.062905 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:14.643956 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:14.643925 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:21.847311 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:21.847261 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:22.000107 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:22.000069 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:22.096844 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:22.096807 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:28.459098 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:28.459061 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:28:32.576866 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:32.576777 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:32.818280 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:32.818247 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:37.454264 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:37.454222 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:38.866090 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:38.866055 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:39.828717 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:39.828684 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv"] Sep 10 16:28:39.850926 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:39.850886 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:43.955409 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:43.955369 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:28:49.810823 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:49.810784 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:53.734835 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:53.734798 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:54.848649 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:54.848602 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" podUID="76abd234-f8ea-4ebe-bfa9-1a595ad92f94" containerName="main" containerID="cri-o://e71d4a246f7ac47cc171ee30d83bae7cedd6775bd3b0a5d9d06b6f2e2372651e" gracePeriod=45 Sep 10 16:28:55.106615 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.106551 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:28:55.284607 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.284570 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-model-cache\") pod \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " Sep 10 16:28:55.284793 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.284616 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-home\") pod \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " Sep 10 16:28:55.284793 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.284727 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dp74z\" (UniqueName: \"kubernetes.io/projected/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-kube-api-access-dp74z\") pod \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " Sep 10 16:28:55.284793 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.284779 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-tmp-dir\") pod \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " Sep 10 16:28:55.284969 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.284810 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-dshm\") pod \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " Sep 10 16:28:55.284969 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.284845 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-tls-certs\") pod \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\" (UID: \"76abd234-f8ea-4ebe-bfa9-1a595ad92f94\") " Sep 10 16:28:55.284969 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.284903 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-model-cache" (OuterVolumeSpecName: "model-cache") pod "76abd234-f8ea-4ebe-bfa9-1a595ad92f94" (UID: "76abd234-f8ea-4ebe-bfa9-1a595ad92f94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:28:55.285124 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.284973 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-home" (OuterVolumeSpecName: "home") pod "76abd234-f8ea-4ebe-bfa9-1a595ad92f94" (UID: "76abd234-f8ea-4ebe-bfa9-1a595ad92f94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:28:55.285124 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.285057 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76abd234-f8ea-4ebe-bfa9-1a595ad92f94" (UID: "76abd234-f8ea-4ebe-bfa9-1a595ad92f94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:28:55.285241 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.285214 2541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-tmp-dir\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:28:55.285241 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.285240 2541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-model-cache\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:28:55.285460 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.285254 2541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-home\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:28:55.287336 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.287293 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-kube-api-access-dp74z" (OuterVolumeSpecName: "kube-api-access-dp74z") pod "76abd234-f8ea-4ebe-bfa9-1a595ad92f94" (UID: "76abd234-f8ea-4ebe-bfa9-1a595ad92f94"). InnerVolumeSpecName "kube-api-access-dp74z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:28:55.287589 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.287563 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-dshm" (OuterVolumeSpecName: "dshm") pod "76abd234-f8ea-4ebe-bfa9-1a595ad92f94" (UID: "76abd234-f8ea-4ebe-bfa9-1a595ad92f94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:28:55.287589 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.287569 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76abd234-f8ea-4ebe-bfa9-1a595ad92f94" (UID: "76abd234-f8ea-4ebe-bfa9-1a595ad92f94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:28:55.386049 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.386015 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp74z\" (UniqueName: \"kubernetes.io/projected/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-kube-api-access-dp74z\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:28:55.386049 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.386044 2541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-dshm\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:28:55.386049 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.386054 2541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76abd234-f8ea-4ebe-bfa9-1a595ad92f94-tls-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:28:55.550975 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.550942 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:28:55.842527 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.842434 2541 generic.go:356] "Generic (PLEG): container finished" podID="76abd234-f8ea-4ebe-bfa9-1a595ad92f94" containerID="e71d4a246f7ac47cc171ee30d83bae7cedd6775bd3b0a5d9d06b6f2e2372651e" exitCode=0 Sep 10 16:28:55.842527 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.842474 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" Sep 10 16:28:55.842527 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.842504 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" event={"ID":"76abd234-f8ea-4ebe-bfa9-1a595ad92f94","Type":"ContainerDied","Data":"e71d4a246f7ac47cc171ee30d83bae7cedd6775bd3b0a5d9d06b6f2e2372651e"} Sep 10 16:28:55.842806 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.842535 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv" event={"ID":"76abd234-f8ea-4ebe-bfa9-1a595ad92f94","Type":"ContainerDied","Data":"9098b1c385e2b1ce3617fee4f3c9649c99720459a7458d812fc632d184732bc9"} Sep 10 16:28:55.842806 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.842552 2541 scope.go:117] "RemoveContainer" containerID="e71d4a246f7ac47cc171ee30d83bae7cedd6775bd3b0a5d9d06b6f2e2372651e" Sep 10 16:28:55.852489 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.852270 2541 scope.go:117] "RemoveContainer" containerID="e71d4a246f7ac47cc171ee30d83bae7cedd6775bd3b0a5d9d06b6f2e2372651e" Sep 10 16:28:55.852729 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:28:55.852584 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e71d4a246f7ac47cc171ee30d83bae7cedd6775bd3b0a5d9d06b6f2e2372651e\": container with ID starting with e71d4a246f7ac47cc171ee30d83bae7cedd6775bd3b0a5d9d06b6f2e2372651e not found: ID does not exist" containerID="e71d4a246f7ac47cc171ee30d83bae7cedd6775bd3b0a5d9d06b6f2e2372651e" Sep 10 16:28:55.852729 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.852612 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e71d4a246f7ac47cc171ee30d83bae7cedd6775bd3b0a5d9d06b6f2e2372651e"} err="failed to get container status \"e71d4a246f7ac47cc171ee30d83bae7cedd6775bd3b0a5d9d06b6f2e2372651e\": rpc error: code = NotFound desc = could not find container \"e71d4a246f7ac47cc171ee30d83bae7cedd6775bd3b0a5d9d06b6f2e2372651e\": container with ID starting with e71d4a246f7ac47cc171ee30d83bae7cedd6775bd3b0a5d9d06b6f2e2372651e not found: ID does not exist" Sep 10 16:28:55.866211 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.866184 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv"] Sep 10 16:28:55.872424 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:55.872400 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b73303/sub-test-model-b73303-kserve-db4549667-qchkv"] Sep 10 16:28:56.980271 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:28:56.980229 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76abd234-f8ea-4ebe-bfa9-1a595ad92f94" path="/var/lib/kubelet/pods/76abd234-f8ea-4ebe-bfa9-1a595ad92f94/volumes" Sep 10 16:29:01.059553 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.059517 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:01.135870 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.135835 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:01.445461 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.445422 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:01.751289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.751207 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt"] Sep 10 16:29:01.751826 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.751790 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cf19c97-1191-4575-9aaa-4a01d89fbb28" containerName="istio-proxy" Sep 10 16:29:01.751826 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.751815 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cf19c97-1191-4575-9aaa-4a01d89fbb28" containerName="istio-proxy" Sep 10 16:29:01.751826 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.751826 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76abd234-f8ea-4ebe-bfa9-1a595ad92f94" containerName="main" Sep 10 16:29:01.752044 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.751836 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="76abd234-f8ea-4ebe-bfa9-1a595ad92f94" containerName="main" Sep 10 16:29:01.752044 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.751930 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cf19c97-1191-4575-9aaa-4a01d89fbb28" containerName="istio-proxy" Sep 10 16:29:01.752044 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.751941 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="76abd234-f8ea-4ebe-bfa9-1a595ad92f94" containerName="main" Sep 10 16:29:01.756497 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.756474 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.758746 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.758724 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ab677e-openshift-default-dockercfg-4k7tr\"" Sep 10 16:29:01.768550 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.768527 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt"] Sep 10 16:29:01.789266 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.789236 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:01.839883 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.839839 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-envoy\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.840128 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.839915 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-token\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.840128 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.839941 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da531cc7-1634-4ec1-86e2-28aeeb94a025-istiod-ca-cert\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.840128 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.839970 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-podinfo\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.840128 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.840099 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-data\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.840405 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.840159 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcwdd\" (UniqueName: \"kubernetes.io/projected/da531cc7-1634-4ec1-86e2-28aeeb94a025-kube-api-access-xcwdd\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.840405 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.840196 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-credential-socket\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.840405 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.840264 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-workload-socket\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.840405 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.840292 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-workload-certs\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.940838 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.940797 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-data\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.941038 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.940851 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcwdd\" (UniqueName: \"kubernetes.io/projected/da531cc7-1634-4ec1-86e2-28aeeb94a025-kube-api-access-xcwdd\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.941038 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.940873 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-credential-socket\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.941038 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.940921 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-workload-socket\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.941038 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.940944 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-workload-certs\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.941038 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.940983 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-envoy\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.941038 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.941030 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-token\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.941393 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.941056 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da531cc7-1634-4ec1-86e2-28aeeb94a025-istiod-ca-cert\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.941393 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.941085 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-podinfo\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.941393 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.941293 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-data\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.941393 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.941357 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-workload-socket\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.941612 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.941411 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-credential-socket\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.941612 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.941490 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-workload-certs\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.942172 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.942140 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da531cc7-1634-4ec1-86e2-28aeeb94a025-istiod-ca-cert\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.943650 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.943626 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-envoy\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.943800 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.943782 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-podinfo\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.961831 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.961806 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-token\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:01.963614 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:01.963591 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcwdd\" (UniqueName: \"kubernetes.io/projected/da531cc7-1634-4ec1-86e2-28aeeb94a025-kube-api-access-xcwdd\") pod \"e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:02.067827 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:02.067739 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:02.082551 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:02.082515 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:02.353605 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:02.353573 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt"] Sep 10 16:29:02.354548 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:29:02.354525 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda531cc7_1634_4ec1_86e2_28aeeb94a025.slice/crio-ba1792c3f105d6aed764b9c278f7fe602b26e29dfad2727d2d4625c328e115a6 WatchSource:0}: Error finding container ba1792c3f105d6aed764b9c278f7fe602b26e29dfad2727d2d4625c328e115a6: Status 404 returned error can't find the container with id ba1792c3f105d6aed764b9c278f7fe602b26e29dfad2727d2d4625c328e115a6 Sep 10 16:29:02.356748 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:02.356713 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:29:02.356845 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:02.356796 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:29:02.356913 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:02.356842 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:29:02.889499 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:02.889464 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" event={"ID":"da531cc7-1634-4ec1-86e2-28aeeb94a025","Type":"ContainerStarted","Data":"8dc8d63a705bbfe3214218d1bc5507fb0ce6b57250a4a15c10ee422612a86115"} Sep 10 16:29:02.889661 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:02.889524 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" event={"ID":"da531cc7-1634-4ec1-86e2-28aeeb94a025","Type":"ContainerStarted","Data":"ba1792c3f105d6aed764b9c278f7fe602b26e29dfad2727d2d4625c328e115a6"} Sep 10 16:29:02.914162 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:02.914108 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" podStartSLOduration=1.9140910770000001 podStartE2EDuration="1.914091077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:29:02.912690622 +0000 UTC m=+1642.435935819" watchObservedRunningTime="2026-09-10 16:29:02.914091077 +0000 UTC m=+1642.437336274" Sep 10 16:29:03.068627 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:03.068591 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:03.073787 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:03.073767 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:03.895258 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:03.895225 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:03.896372 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:03.896353 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:04.951224 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:04.951156 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:05.060739 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:05.060706 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:06.253417 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:06.253379 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:06.364140 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:06.364105 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:06.466998 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:06.466964 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:06.911089 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:06.911053 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:07.010043 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:07.010006 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:07.446770 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:07.446736 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:08.416389 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:08.416356 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:11.974912 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:11.974867 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:29:18.115167 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.115134 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs"] Sep 10 16:29:18.115702 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.115683 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cf19c97-1191-4575-9aaa-4a01d89fbb28" containerName="istio-proxy" Sep 10 16:29:18.115702 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.115704 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cf19c97-1191-4575-9aaa-4a01d89fbb28" containerName="istio-proxy" Sep 10 16:29:18.115840 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.115765 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cf19c97-1191-4575-9aaa-4a01d89fbb28" containerName="istio-proxy" Sep 10 16:29:18.119771 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.119741 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:18.125026 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.124892 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b73303\"/\"rate-test-model-b73303-kserve-self-signed-certs\"" Sep 10 16:29:18.125026 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.124931 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b73303\"/\"kube-root-ca.crt\"" Sep 10 16:29:18.125026 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.124957 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b73303\"/\"openshift-service-ca.crt\"" Sep 10 16:29:18.125336 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.125275 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b73303\"/\"default-dockercfg-2jzpz\"" Sep 10 16:29:18.137109 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.137086 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs"] Sep 10 16:29:18.212897 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.212863 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-dshm\") pod \"rate-test-model-b73303-kserve-895747d75-g8ffs\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:18.212897 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.212898 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-tmp-dir\") pod \"rate-test-model-b73303-kserve-895747d75-g8ffs\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:18.213113 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.212930 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-home\") pod \"rate-test-model-b73303-kserve-895747d75-g8ffs\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:18.213113 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.212951 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-tls-certs\") pod \"rate-test-model-b73303-kserve-895747d75-g8ffs\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:18.213113 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.212970 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glb8l\" (UniqueName: \"kubernetes.io/projected/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-kube-api-access-glb8l\") pod \"rate-test-model-b73303-kserve-895747d75-g8ffs\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:18.213113 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.213084 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-model-cache\") pod \"rate-test-model-b73303-kserve-895747d75-g8ffs\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:18.314571 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.314532 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-dshm\") pod \"rate-test-model-b73303-kserve-895747d75-g8ffs\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:18.314571 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.314571 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-tmp-dir\") pod \"rate-test-model-b73303-kserve-895747d75-g8ffs\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:18.314832 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.314598 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-home\") pod \"rate-test-model-b73303-kserve-895747d75-g8ffs\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:18.314832 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.314620 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-tls-certs\") pod \"rate-test-model-b73303-kserve-895747d75-g8ffs\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:18.314832 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.314643 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glb8l\" (UniqueName: \"kubernetes.io/projected/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-kube-api-access-glb8l\") pod \"rate-test-model-b73303-kserve-895747d75-g8ffs\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:18.314832 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.314733 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-model-cache\") pod \"rate-test-model-b73303-kserve-895747d75-g8ffs\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:18.315078 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.315055 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-tmp-dir\") pod \"rate-test-model-b73303-kserve-895747d75-g8ffs\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:18.315127 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.315080 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-home\") pod \"rate-test-model-b73303-kserve-895747d75-g8ffs\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:18.315127 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.315105 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-model-cache\") pod \"rate-test-model-b73303-kserve-895747d75-g8ffs\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:18.317038 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.317017 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-dshm\") pod \"rate-test-model-b73303-kserve-895747d75-g8ffs\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:18.317166 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.317149 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-tls-certs\") pod \"rate-test-model-b73303-kserve-895747d75-g8ffs\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:18.324561 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.324539 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glb8l\" (UniqueName: \"kubernetes.io/projected/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-kube-api-access-glb8l\") pod \"rate-test-model-b73303-kserve-895747d75-g8ffs\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:18.432170 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.432115 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:18.724495 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.724464 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs"] Sep 10 16:29:18.725819 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:29:18.725794 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c85386e_6f75_4bfd_aa2a_5bd010307b3f.slice/crio-2c4a30f4f74313ddbcdd6effb964e8c2fed58b1f9eb6b967d7f98db248424a7c WatchSource:0}: Error finding container 2c4a30f4f74313ddbcdd6effb964e8c2fed58b1f9eb6b967d7f98db248424a7c: Status 404 returned error can't find the container with id 2c4a30f4f74313ddbcdd6effb964e8c2fed58b1f9eb6b967d7f98db248424a7c Sep 10 16:29:18.968018 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.967986 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:18.992277 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.992133 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" event={"ID":"0c85386e-6f75-4bfd-aa2a-5bd010307b3f","Type":"ContainerStarted","Data":"c10cbbe9acb102fca75fee5ebecbbc6aadca2a8768d06397f04cc65511ff4320"} Sep 10 16:29:18.992277 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:18.992185 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" event={"ID":"0c85386e-6f75-4bfd-aa2a-5bd010307b3f","Type":"ContainerStarted","Data":"2c4a30f4f74313ddbcdd6effb964e8c2fed58b1f9eb6b967d7f98db248424a7c"} Sep 10 16:29:19.014719 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:19.014667 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" podStartSLOduration=1.014649403 podStartE2EDuration="1.014649403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:29:19.012116425 +0000 UTC m=+1658.535361629" watchObservedRunningTime="2026-09-10 16:29:19.014649403 +0000 UTC m=+1658.537894600" Sep 10 16:29:21.457130 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:21.457060 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:21.534090 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:21.534040 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:23.019173 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:23.019141 2541 generic.go:356] "Generic (PLEG): container finished" podID="da531cc7-1634-4ec1-86e2-28aeeb94a025" containerID="8dc8d63a705bbfe3214218d1bc5507fb0ce6b57250a4a15c10ee422612a86115" exitCode=0 Sep 10 16:29:23.019619 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:23.019220 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" event={"ID":"da531cc7-1634-4ec1-86e2-28aeeb94a025","Type":"ContainerDied","Data":"8dc8d63a705bbfe3214218d1bc5507fb0ce6b57250a4a15c10ee422612a86115"} Sep 10 16:29:23.019681 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:23.019656 2541 scope.go:117] "RemoveContainer" containerID="8dc8d63a705bbfe3214218d1bc5507fb0ce6b57250a4a15c10ee422612a86115" Sep 10 16:29:23.020827 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:23.020800 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:29:23.020882 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:23.020870 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:29:23.020916 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:23.020908 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:29:24.028602 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:24.028563 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" event={"ID":"da531cc7-1634-4ec1-86e2-28aeeb94a025","Type":"ContainerStarted","Data":"fae95858fc1456438959cd87f4c871965b5a59930f52d5204752c377f8fff3c1"} Sep 10 16:29:24.068325 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:24.068256 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:24.068542 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:24.068516 2541 patch_prober.go:28] interesting pod/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 10 16:29:24.068593 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:24.068573 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" podUID="da531cc7-1634-4ec1-86e2-28aeeb94a025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 10 16:29:25.068725 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:25.068687 2541 patch_prober.go:28] interesting pod/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 10 16:29:25.069110 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:25.068745 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" podUID="da531cc7-1634-4ec1-86e2-28aeeb94a025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 10 16:29:26.068902 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:26.068872 2541 patch_prober.go:28] interesting pod/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 10 16:29:26.069292 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:26.068924 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" podUID="da531cc7-1634-4ec1-86e2-28aeeb94a025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 10 16:29:27.069051 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:27.069015 2541 patch_prober.go:28] interesting pod/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 10 16:29:27.069494 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:27.069081 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" podUID="da531cc7-1634-4ec1-86e2-28aeeb94a025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 10 16:29:28.068770 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:28.068729 2541 patch_prober.go:28] interesting pod/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 10 16:29:28.068937 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:28.068796 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" podUID="da531cc7-1634-4ec1-86e2-28aeeb94a025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 10 16:29:28.432717 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:28.432679 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:28.432717 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:28.432729 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:28.445552 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:28.445518 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:29.069166 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:29.069136 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:29:29.069166 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:29.069155 2541 patch_prober.go:28] interesting pod/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 10 16:29:29.069446 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:29.069221 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" podUID="da531cc7-1634-4ec1-86e2-28aeeb94a025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 10 16:29:29.957832 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:29.957792 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:30.072702 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:30.072661 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:30.073089 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:30.073066 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:30.073899 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:30.073879 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:29:30.459085 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:30.459048 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:30.771531 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:30.771445 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh"] Sep 10 16:29:30.775762 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:30.775727 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh"] Sep 10 16:29:30.775958 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:30.775828 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:30.777041 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:30.777015 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:30.778042 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:30.778019 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-cde4a2-openshift-default-dockercfg-wglgk\"" Sep 10 16:29:30.923737 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:30.923697 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:30.952380 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:30.952331 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-credential-socket\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:30.952545 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:30.952391 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-envoy\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:30.952545 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:30.952420 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-workload-socket\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:30.952545 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:30.952438 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-podinfo\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:30.952545 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:30.952490 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfxwk\" (UniqueName: \"kubernetes.io/projected/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-kube-api-access-nfxwk\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:30.952545 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:30.952506 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istiod-ca-cert\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:30.952545 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:30.952523 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-token\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:30.952771 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:30.952596 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-workload-certs\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:30.952771 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:30.952692 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-data\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:31.053829 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.053722 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-credential-socket\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:31.053829 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.053776 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-envoy\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:31.054396 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.053909 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-workload-socket\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:31.054396 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.053955 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-podinfo\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:31.054396 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.054011 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfxwk\" (UniqueName: \"kubernetes.io/projected/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-kube-api-access-nfxwk\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:31.054396 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.054035 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istiod-ca-cert\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:31.054396 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.054064 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-token\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:31.054396 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.054132 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-workload-certs\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:31.054396 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.054140 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-credential-socket\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:31.054396 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.054245 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-data\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:31.054396 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.054256 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-workload-socket\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:31.054755 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.054483 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-workload-certs\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:31.054755 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.054530 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-data\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:31.054877 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.054811 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istiod-ca-cert\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:31.056438 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.056420 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-envoy\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:31.056735 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.056719 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-podinfo\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:31.066391 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.066362 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-token\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:31.066582 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.066564 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfxwk\" (UniqueName: \"kubernetes.io/projected/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-kube-api-access-nfxwk\") pod \"e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:31.090053 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.090001 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:31.383568 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.383533 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh"] Sep 10 16:29:31.385023 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:29:31.384995 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dcdaad8_c65f_4fc3_b952_1c4521feeae5.slice/crio-fcbaa00c396f507e9516c50c86d88cf7553e19f2ceefe2d4caf255b2608e35d5 WatchSource:0}: Error finding container fcbaa00c396f507e9516c50c86d88cf7553e19f2ceefe2d4caf255b2608e35d5: Status 404 returned error can't find the container with id fcbaa00c396f507e9516c50c86d88cf7553e19f2ceefe2d4caf255b2608e35d5 Sep 10 16:29:31.388058 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.388016 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:29:31.388156 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.388110 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:29:31.388196 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.388160 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:29:31.957147 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:31.957100 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:29:32.078284 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:32.078245 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" event={"ID":"0dcdaad8-c65f-4fc3-b952-1c4521feeae5","Type":"ContainerStarted","Data":"a27820173fe7ec34d9f9900a4aad4f3095a530cc823356754b83c7a0ddde0978"} Sep 10 16:29:32.078809 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:32.078320 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" event={"ID":"0dcdaad8-c65f-4fc3-b952-1c4521feeae5","Type":"ContainerStarted","Data":"fcbaa00c396f507e9516c50c86d88cf7553e19f2ceefe2d4caf255b2608e35d5"} Sep 10 16:29:32.091106 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:32.091070 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:32.096858 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:32.096828 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:32.107415 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:32.107354 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" podStartSLOduration=2.107335644 podStartE2EDuration="2.107335644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:29:32.104204489 +0000 UTC m=+1671.627449687" watchObservedRunningTime="2026-09-10 16:29:32.107335644 +0000 UTC m=+1671.630580841" Sep 10 16:29:33.087092 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:33.087055 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:33.088224 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:33.088202 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:34.027038 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:34.027007 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:34.217077 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:34.217038 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:35.317038 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:35.317005 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:35.390139 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:35.390105 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:35.487660 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:35.487625 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:36.010449 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:36.010413 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:36.116882 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:36.116850 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:52.205807 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:52.205770 2541 generic.go:356] "Generic (PLEG): container finished" podID="0dcdaad8-c65f-4fc3-b952-1c4521feeae5" containerID="a27820173fe7ec34d9f9900a4aad4f3095a530cc823356754b83c7a0ddde0978" exitCode=0 Sep 10 16:29:52.206253 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:52.205833 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" event={"ID":"0dcdaad8-c65f-4fc3-b952-1c4521feeae5","Type":"ContainerDied","Data":"a27820173fe7ec34d9f9900a4aad4f3095a530cc823356754b83c7a0ddde0978"} Sep 10 16:29:52.206330 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:52.206257 2541 scope.go:117] "RemoveContainer" containerID="a27820173fe7ec34d9f9900a4aad4f3095a530cc823356754b83c7a0ddde0978" Sep 10 16:29:52.207668 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:52.207634 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:29:52.207748 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:52.207712 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:29:52.207786 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:52.207747 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:29:53.178018 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:53.177977 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:29:53.220051 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:53.219983 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" event={"ID":"0dcdaad8-c65f-4fc3-b952-1c4521feeae5","Type":"ContainerStarted","Data":"13eca48ade8b780096521480b72fcc73641e2b911a083cd46ffb0bd238e55719"} Sep 10 16:29:54.091229 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:54.091189 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:54.091624 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:54.091601 2541 patch_prober.go:28] interesting pod/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 10 16:29:54.091762 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:54.091739 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" podUID="0dcdaad8-c65f-4fc3-b952-1c4521feeae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 10 16:29:55.091045 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:55.090993 2541 patch_prober.go:28] interesting pod/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 10 16:29:55.091600 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:55.091097 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" podUID="0dcdaad8-c65f-4fc3-b952-1c4521feeae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 10 16:29:56.090708 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:56.090658 2541 patch_prober.go:28] interesting pod/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 10 16:29:56.090935 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:56.090747 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" podUID="0dcdaad8-c65f-4fc3-b952-1c4521feeae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 10 16:29:57.091135 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:57.091080 2541 patch_prober.go:28] interesting pod/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 10 16:29:57.091693 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:57.091170 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" podUID="0dcdaad8-c65f-4fc3-b952-1c4521feeae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 10 16:29:58.090729 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:58.090678 2541 patch_prober.go:28] interesting pod/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 10 16:29:58.091052 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:58.091023 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" podUID="0dcdaad8-c65f-4fc3-b952-1c4521feeae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 10 16:29:59.001076 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:59.001035 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:59.281783 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:59.281663 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:59.282644 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:59.282612 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:29:59.285024 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:29:59.284994 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:30:00.179374 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:00.179332 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab677e-29817630-6wn8s"] Sep 10 16:30:00.184412 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:00.184381 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-219fd6-29817630-sp7jf"] Sep 10 16:30:00.184587 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:00.184537 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab677e-29817630-6wn8s" Sep 10 16:30:00.187730 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:00.187698 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l2dj2\"" Sep 10 16:30:00.188465 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:00.188437 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b73303-29817630-84psd"] Sep 10 16:30:00.188715 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:00.188699 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-219fd6-29817630-sp7jf" Sep 10 16:30:00.192783 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:00.192754 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b73303-29817630-84psd" Sep 10 16:30:00.197799 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:00.197757 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab677e-29817630-6wn8s"] Sep 10 16:30:00.202010 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:00.201966 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-219fd6-29817630-sp7jf"] Sep 10 16:30:00.203489 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:00.203464 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab677e-29817630-6wn8s" Sep 10 16:30:00.217865 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:00.217836 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-219fd6-29817630-sp7jf" Sep 10 16:30:00.219835 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:00.219804 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cde4a2-29817630-48fnl"] Sep 10 16:30:00.230631 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:00.230590 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b73303-29817630-84psd"] Sep 10 16:30:00.230808 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:00.230709 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cde4a2-29817630-48fnl" Sep 10 16:30:00.238417 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:00.238366 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cde4a2-29817630-48fnl"] Sep 10 16:30:00.272231 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:00.272202 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b73303-29817630-84psd" Sep 10 16:30:00.286275 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:00.283659 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cde4a2-29817630-48fnl" Sep 10 16:30:00.509567 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:00.509537 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab677e-29817630-6wn8s"] Sep 10 16:30:00.511227 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:30:00.511197 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b53b71_3021_4571_bb91_5cd76805da64.slice/crio-aa4bfb81cb01b946a888d2c8347003637d4c40dbe33bc3a4745f69b27d84435a WatchSource:0}: Error finding container aa4bfb81cb01b946a888d2c8347003637d4c40dbe33bc3a4745f69b27d84435a: Status 404 returned error can't find the container with id aa4bfb81cb01b946a888d2c8347003637d4c40dbe33bc3a4745f69b27d84435a Sep 10 16:30:00.548863 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:00.548837 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-219fd6-29817630-sp7jf"] Sep 10 16:30:00.550637 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:30:00.550615 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515fda7a_e2bb_4542_a0f1_71f019b374fd.slice/crio-88dcf7b9605601dd176f863b88a5be61304c3d0f7b9c11279f2df600527339f7 WatchSource:0}: Error finding container 88dcf7b9605601dd176f863b88a5be61304c3d0f7b9c11279f2df600527339f7: Status 404 returned error can't find the container with id 88dcf7b9605601dd176f863b88a5be61304c3d0f7b9c11279f2df600527339f7 Sep 10 16:30:00.596407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:00.596359 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b73303-29817630-84psd"] Sep 10 16:30:00.599455 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:00.599420 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cde4a2-29817630-48fnl"] Sep 10 16:30:00.599715 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:30:00.599690 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e54c8c_d5e2_430d_bce5_7596c8fd9bc0.slice/crio-675d44cd832bdd01f8bf23d5a27cf9e991a57bafcab6a104125eb2d84f31be66 WatchSource:0}: Error finding container 675d44cd832bdd01f8bf23d5a27cf9e991a57bafcab6a104125eb2d84f31be66: Status 404 returned error can't find the container with id 675d44cd832bdd01f8bf23d5a27cf9e991a57bafcab6a104125eb2d84f31be66 Sep 10 16:30:00.600186 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:30:00.600162 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f6ee1e_2027_44e5_ace6_cf30083ffcde.slice/crio-43eb59c5a83af24692203bcf6693b281668001468c02714ecb68957f341c03a1 WatchSource:0}: Error finding container 43eb59c5a83af24692203bcf6693b281668001468c02714ecb68957f341c03a1: Status 404 returned error can't find the container with id 43eb59c5a83af24692203bcf6693b281668001468c02714ecb68957f341c03a1 Sep 10 16:30:01.283559 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:01.283452 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-219fd6-29817630-sp7jf" event={"ID":"515fda7a-e2bb-4542-a0f1-71f019b374fd","Type":"ContainerStarted","Data":"613bbaca2988075963ddefc339b8258c282107abcd358a8d2de2cce12d72ae00"} Sep 10 16:30:01.283559 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:01.283517 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-219fd6-29817630-sp7jf" event={"ID":"515fda7a-e2bb-4542-a0f1-71f019b374fd","Type":"ContainerStarted","Data":"88dcf7b9605601dd176f863b88a5be61304c3d0f7b9c11279f2df600527339f7"} Sep 10 16:30:01.285538 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:01.285502 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab677e-29817630-6wn8s" event={"ID":"c2b53b71-3021-4571-bb91-5cd76805da64","Type":"ContainerStarted","Data":"5124aced9dd0cd8cebcbe05989359f70187d5f4f6c462ff4dea110f45da3c092"} Sep 10 16:30:01.285707 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:01.285572 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab677e-29817630-6wn8s" event={"ID":"c2b53b71-3021-4571-bb91-5cd76805da64","Type":"ContainerStarted","Data":"aa4bfb81cb01b946a888d2c8347003637d4c40dbe33bc3a4745f69b27d84435a"} Sep 10 16:30:01.287372 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:01.287338 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b73303-29817630-84psd" event={"ID":"86e54c8c-d5e2-430d-bce5-7596c8fd9bc0","Type":"ContainerStarted","Data":"7f4c8327e16cb1cf29e0a477505bacd3590925914f1cbc6be2491082a1605b0f"} Sep 10 16:30:01.287559 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:01.287398 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b73303-29817630-84psd" event={"ID":"86e54c8c-d5e2-430d-bce5-7596c8fd9bc0","Type":"ContainerStarted","Data":"675d44cd832bdd01f8bf23d5a27cf9e991a57bafcab6a104125eb2d84f31be66"} Sep 10 16:30:01.289383 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:01.289354 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cde4a2-29817630-48fnl" event={"ID":"d2f6ee1e-2027-44e5-ace6-cf30083ffcde","Type":"ContainerStarted","Data":"52f09693e7b206cb3c150febed62648b805797f8493f3198337cd416cdc304ea"} Sep 10 16:30:01.289553 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:01.289401 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cde4a2-29817630-48fnl" event={"ID":"d2f6ee1e-2027-44e5-ace6-cf30083ffcde","Type":"ContainerStarted","Data":"43eb59c5a83af24692203bcf6693b281668001468c02714ecb68957f341c03a1"} Sep 10 16:30:01.302269 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:01.302210 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-219fd6-29817630-sp7jf" podStartSLOduration=1.302189647 podStartE2EDuration="1.302189647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:30:01.299809558 +0000 UTC m=+1700.823054780" watchObservedRunningTime="2026-09-10 16:30:01.302189647 +0000 UTC m=+1700.825434844" Sep 10 16:30:01.318884 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:01.318824 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b73303-29817630-84psd" podStartSLOduration=1.318807064 podStartE2EDuration="1.318807064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:30:01.315653566 +0000 UTC m=+1700.838898764" watchObservedRunningTime="2026-09-10 16:30:01.318807064 +0000 UTC m=+1700.842052261" Sep 10 16:30:01.332690 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:01.332638 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cde4a2-29817630-48fnl" podStartSLOduration=1.332620097 podStartE2EDuration="1.332620097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:30:01.330404622 +0000 UTC m=+1700.853649819" watchObservedRunningTime="2026-09-10 16:30:01.332620097 +0000 UTC m=+1700.855865333" Sep 10 16:30:01.347383 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:01.347330 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab677e-29817630-6wn8s" podStartSLOduration=1.347288293 podStartE2EDuration="1.347288293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:30:01.345098313 +0000 UTC m=+1700.868343523" watchObservedRunningTime="2026-09-10 16:30:01.347288293 +0000 UTC m=+1700.870533489" Sep 10 16:30:02.238864 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:02.238820 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:02.301649 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:02.301563 2541 generic.go:356] "Generic (PLEG): container finished" podID="86e54c8c-d5e2-430d-bce5-7596c8fd9bc0" containerID="7f4c8327e16cb1cf29e0a477505bacd3590925914f1cbc6be2491082a1605b0f" exitCode=0 Sep 10 16:30:02.301649 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:02.301619 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b73303-29817630-84psd" event={"ID":"86e54c8c-d5e2-430d-bce5-7596c8fd9bc0","Type":"ContainerDied","Data":"7f4c8327e16cb1cf29e0a477505bacd3590925914f1cbc6be2491082a1605b0f"} Sep 10 16:30:02.306412 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:02.306362 2541 generic.go:356] "Generic (PLEG): container finished" podID="d2f6ee1e-2027-44e5-ace6-cf30083ffcde" containerID="52f09693e7b206cb3c150febed62648b805797f8493f3198337cd416cdc304ea" exitCode=0 Sep 10 16:30:02.306660 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:02.306631 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cde4a2-29817630-48fnl" event={"ID":"d2f6ee1e-2027-44e5-ace6-cf30083ffcde","Type":"ContainerDied","Data":"52f09693e7b206cb3c150febed62648b805797f8493f3198337cd416cdc304ea"} Sep 10 16:30:02.314515 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:02.314473 2541 generic.go:356] "Generic (PLEG): container finished" podID="515fda7a-e2bb-4542-a0f1-71f019b374fd" containerID="613bbaca2988075963ddefc339b8258c282107abcd358a8d2de2cce12d72ae00" exitCode=0 Sep 10 16:30:02.314720 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:02.314561 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-219fd6-29817630-sp7jf" event={"ID":"515fda7a-e2bb-4542-a0f1-71f019b374fd","Type":"ContainerDied","Data":"613bbaca2988075963ddefc339b8258c282107abcd358a8d2de2cce12d72ae00"} Sep 10 16:30:02.316564 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:02.316526 2541 generic.go:356] "Generic (PLEG): container finished" podID="c2b53b71-3021-4571-bb91-5cd76805da64" containerID="5124aced9dd0cd8cebcbe05989359f70187d5f4f6c462ff4dea110f45da3c092" exitCode=0 Sep 10 16:30:02.316701 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:02.316577 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab677e-29817630-6wn8s" event={"ID":"c2b53b71-3021-4571-bb91-5cd76805da64","Type":"ContainerDied","Data":"5124aced9dd0cd8cebcbe05989359f70187d5f4f6c462ff4dea110f45da3c092"} Sep 10 16:30:03.518706 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:03.518672 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-219fd6-29817630-sp7jf" Sep 10 16:30:03.520975 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:03.520941 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b73303-29817630-84psd" Sep 10 16:30:03.574450 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:03.574377 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cde4a2-29817630-48fnl" Sep 10 16:30:03.578429 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:03.578400 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab677e-29817630-6wn8s" Sep 10 16:30:04.330910 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:04.330883 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-219fd6-29817630-sp7jf" Sep 10 16:30:04.331134 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:04.330914 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-219fd6-29817630-sp7jf" event={"ID":"515fda7a-e2bb-4542-a0f1-71f019b374fd","Type":"ContainerDied","Data":"88dcf7b9605601dd176f863b88a5be61304c3d0f7b9c11279f2df600527339f7"} Sep 10 16:30:04.331134 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:04.330960 2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88dcf7b9605601dd176f863b88a5be61304c3d0f7b9c11279f2df600527339f7" Sep 10 16:30:04.332596 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:04.332569 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab677e-29817630-6wn8s" Sep 10 16:30:04.332596 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:04.332577 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab677e-29817630-6wn8s" event={"ID":"c2b53b71-3021-4571-bb91-5cd76805da64","Type":"ContainerDied","Data":"aa4bfb81cb01b946a888d2c8347003637d4c40dbe33bc3a4745f69b27d84435a"} Sep 10 16:30:04.332790 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:04.332617 2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa4bfb81cb01b946a888d2c8347003637d4c40dbe33bc3a4745f69b27d84435a" Sep 10 16:30:04.334358 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:04.334321 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b73303-29817630-84psd" event={"ID":"86e54c8c-d5e2-430d-bce5-7596c8fd9bc0","Type":"ContainerDied","Data":"675d44cd832bdd01f8bf23d5a27cf9e991a57bafcab6a104125eb2d84f31be66"} Sep 10 16:30:04.334358 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:04.334358 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b73303-29817630-84psd" Sep 10 16:30:04.334584 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:04.334378 2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="675d44cd832bdd01f8bf23d5a27cf9e991a57bafcab6a104125eb2d84f31be66" Sep 10 16:30:04.336146 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:04.336118 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cde4a2-29817630-48fnl" Sep 10 16:30:04.336327 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:04.336146 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cde4a2-29817630-48fnl" event={"ID":"d2f6ee1e-2027-44e5-ace6-cf30083ffcde","Type":"ContainerDied","Data":"43eb59c5a83af24692203bcf6693b281668001468c02714ecb68957f341c03a1"} Sep 10 16:30:04.336327 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:04.336201 2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43eb59c5a83af24692203bcf6693b281668001468c02714ecb68957f341c03a1" Sep 10 16:30:05.700821 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:05.700790 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:12.187966 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:12.187931 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:15.069165 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:15.069118 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:30:18.097149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:18.097106 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab677e-29817630-6wn8s"] Sep 10 16:30:18.099158 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:18.099124 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:18.103145 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:18.103110 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab677e-29817630-6wn8s"] Sep 10 16:30:18.225432 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:18.225398 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:18.369850 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:18.369757 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:18.516645 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:18.516579 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:18.866043 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:18.866001 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:18.980515 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:18.980478 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2b53b71-3021-4571-bb91-5cd76805da64" path="/var/lib/kubelet/pods/c2b53b71-3021-4571-bb91-5cd76805da64/volumes" Sep 10 16:30:22.428610 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:22.428553 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:23.174477 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:23.174434 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs"] Sep 10 16:30:23.200226 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:23.200175 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:23.626430 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:23.626397 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:23.967275 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:23.967181 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:24.424912 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:24.424883 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:25.459171 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:25.459140 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:25.559280 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:25.559245 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:25.724180 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:25.724088 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt"] Sep 10 16:30:25.724426 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:25.724377 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" podUID="da531cc7-1634-4ec1-86e2-28aeeb94a025" containerName="istio-proxy" containerID="cri-o://fae95858fc1456438959cd87f4c871965b5a59930f52d5204752c377f8fff3c1" gracePeriod=30 Sep 10 16:30:25.778090 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:25.778048 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:26.093567 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.093475 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cde4a2-zjzkh"] Sep 10 16:30:26.093943 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.093928 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f6ee1e-2027-44e5-ace6-cf30083ffcde" containerName="cleanup" Sep 10 16:30:26.093993 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.093945 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f6ee1e-2027-44e5-ace6-cf30083ffcde" containerName="cleanup" Sep 10 16:30:26.093993 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.093961 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86e54c8c-d5e2-430d-bce5-7596c8fd9bc0" containerName="cleanup" Sep 10 16:30:26.093993 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.093966 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="86e54c8c-d5e2-430d-bce5-7596c8fd9bc0" containerName="cleanup" Sep 10 16:30:26.093993 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.093979 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="515fda7a-e2bb-4542-a0f1-71f019b374fd" containerName="cleanup" Sep 10 16:30:26.093993 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.093985 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="515fda7a-e2bb-4542-a0f1-71f019b374fd" containerName="cleanup" Sep 10 16:30:26.094149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.093996 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2b53b71-3021-4571-bb91-5cd76805da64" containerName="cleanup" Sep 10 16:30:26.094149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.094001 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2b53b71-3021-4571-bb91-5cd76805da64" containerName="cleanup" Sep 10 16:30:26.094149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.094066 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="515fda7a-e2bb-4542-a0f1-71f019b374fd" containerName="cleanup" Sep 10 16:30:26.094149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.094075 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f6ee1e-2027-44e5-ace6-cf30083ffcde" containerName="cleanup" Sep 10 16:30:26.094149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.094081 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2b53b71-3021-4571-bb91-5cd76805da64" containerName="cleanup" Sep 10 16:30:26.094149 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.094086 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="86e54c8c-d5e2-430d-bce5-7596c8fd9bc0" containerName="cleanup" Sep 10 16:30:26.098531 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.098506 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cde4a2-zjzkh" Sep 10 16:30:26.100546 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.100522 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:30:26.100689 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.100672 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l2dj2\"" Sep 10 16:30:26.105282 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.105248 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cde4a2-zjzkh"] Sep 10 16:30:26.216506 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.216471 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/465e85db-ee02-44e1-b063-2aabec5c113e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cde4a2-zjzkh\" (UID: \"465e85db-ee02-44e1-b063-2aabec5c113e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cde4a2-zjzkh" Sep 10 16:30:26.216747 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.216727 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:26.317875 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.317836 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/465e85db-ee02-44e1-b063-2aabec5c113e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cde4a2-zjzkh\" (UID: \"465e85db-ee02-44e1-b063-2aabec5c113e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cde4a2-zjzkh" Sep 10 16:30:26.318532 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.318510 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/465e85db-ee02-44e1-b063-2aabec5c113e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cde4a2-zjzkh\" (UID: \"465e85db-ee02-44e1-b063-2aabec5c113e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cde4a2-zjzkh" Sep 10 16:30:26.411437 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.411393 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cde4a2-zjzkh" Sep 10 16:30:26.697682 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.697648 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cde4a2-zjzkh"] Sep 10 16:30:26.698848 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:30:26.698819 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465e85db_ee02_44e1_b063_2aabec5c113e.slice/crio-deccaa929906aaab34cc0c675f255b071b04dddd05aea358e9138f2cdd8a7187 WatchSource:0}: Error finding container deccaa929906aaab34cc0c675f255b071b04dddd05aea358e9138f2cdd8a7187: Status 404 returned error can't find the container with id deccaa929906aaab34cc0c675f255b071b04dddd05aea358e9138f2cdd8a7187 Sep 10 16:30:26.724777 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.724739 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:26.822521 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:26.822489 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:27.497547 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:27.497511 2541 generic.go:356] "Generic (PLEG): container finished" podID="465e85db-ee02-44e1-b063-2aabec5c113e" containerID="f35fb5725170e48f38c7c284a88658e9f4b473755221fd55366e6018d4b87d2c" exitCode=0 Sep 10 16:30:27.497749 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:27.497596 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cde4a2-zjzkh" event={"ID":"465e85db-ee02-44e1-b063-2aabec5c113e","Type":"ContainerDied","Data":"f35fb5725170e48f38c7c284a88658e9f4b473755221fd55366e6018d4b87d2c"} Sep 10 16:30:27.497749 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:27.497653 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cde4a2-zjzkh" event={"ID":"465e85db-ee02-44e1-b063-2aabec5c113e","Type":"ContainerStarted","Data":"deccaa929906aaab34cc0c675f255b071b04dddd05aea358e9138f2cdd8a7187"} Sep 10 16:30:27.937438 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:27.937400 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:28.058467 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:28.058435 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:28.176523 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:28.176483 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:28.568561 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:28.568504 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:28.660093 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:28.660054 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cde4a2-zjzkh" Sep 10 16:30:28.708390 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:28.708359 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:28.744822 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:28.744776 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/465e85db-ee02-44e1-b063-2aabec5c113e-maas-api-service-ca\") pod \"465e85db-ee02-44e1-b063-2aabec5c113e\" (UID: \"465e85db-ee02-44e1-b063-2aabec5c113e\") " Sep 10 16:30:28.745137 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:28.745109 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/465e85db-ee02-44e1-b063-2aabec5c113e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "465e85db-ee02-44e1-b063-2aabec5c113e" (UID: "465e85db-ee02-44e1-b063-2aabec5c113e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:30:28.861389 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:28.849438 2541 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/465e85db-ee02-44e1-b063-2aabec5c113e-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:29.513538 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:29.513507 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cde4a2-zjzkh" Sep 10 16:30:29.513953 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:29.513545 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cde4a2-zjzkh" event={"ID":"465e85db-ee02-44e1-b063-2aabec5c113e","Type":"ContainerDied","Data":"deccaa929906aaab34cc0c675f255b071b04dddd05aea358e9138f2cdd8a7187"} Sep 10 16:30:29.513953 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:29.513605 2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="deccaa929906aaab34cc0c675f255b071b04dddd05aea358e9138f2cdd8a7187" Sep 10 16:30:30.076662 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:30.076617 2541 patch_prober.go:28] interesting pod/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 16:30:30.076860 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:30.076697 2541 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" podUID="da531cc7-1634-4ec1-86e2-28aeeb94a025" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 16:30:31.024182 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.024146 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:30:31.185903 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.185859 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-podinfo\") pod \"da531cc7-1634-4ec1-86e2-28aeeb94a025\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " Sep 10 16:30:31.185903 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.185909 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-envoy\") pod \"da531cc7-1634-4ec1-86e2-28aeeb94a025\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " Sep 10 16:30:31.186345 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.185947 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-workload-certs\") pod \"da531cc7-1634-4ec1-86e2-28aeeb94a025\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " Sep 10 16:30:31.186345 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.185989 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-credential-socket\") pod \"da531cc7-1634-4ec1-86e2-28aeeb94a025\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " Sep 10 16:30:31.186345 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.186027 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-data\") pod \"da531cc7-1634-4ec1-86e2-28aeeb94a025\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " Sep 10 16:30:31.186345 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.186069 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da531cc7-1634-4ec1-86e2-28aeeb94a025-istiod-ca-cert\") pod \"da531cc7-1634-4ec1-86e2-28aeeb94a025\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " Sep 10 16:30:31.186345 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.186108 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-workload-socket\") pod \"da531cc7-1634-4ec1-86e2-28aeeb94a025\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " Sep 10 16:30:31.186345 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.186140 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-token\") pod \"da531cc7-1634-4ec1-86e2-28aeeb94a025\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " Sep 10 16:30:31.186345 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.186208 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcwdd\" (UniqueName: \"kubernetes.io/projected/da531cc7-1634-4ec1-86e2-28aeeb94a025-kube-api-access-xcwdd\") pod \"da531cc7-1634-4ec1-86e2-28aeeb94a025\" (UID: \"da531cc7-1634-4ec1-86e2-28aeeb94a025\") " Sep 10 16:30:31.186726 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.186383 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "da531cc7-1634-4ec1-86e2-28aeeb94a025" (UID: "da531cc7-1634-4ec1-86e2-28aeeb94a025"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:31.186726 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.186573 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "da531cc7-1634-4ec1-86e2-28aeeb94a025" (UID: "da531cc7-1634-4ec1-86e2-28aeeb94a025"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:31.186726 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.186589 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "da531cc7-1634-4ec1-86e2-28aeeb94a025" (UID: "da531cc7-1634-4ec1-86e2-28aeeb94a025"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:31.186726 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.186400 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-data" (OuterVolumeSpecName: "istio-data") pod "da531cc7-1634-4ec1-86e2-28aeeb94a025" (UID: "da531cc7-1634-4ec1-86e2-28aeeb94a025"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:31.186726 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.186599 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da531cc7-1634-4ec1-86e2-28aeeb94a025-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "da531cc7-1634-4ec1-86e2-28aeeb94a025" (UID: "da531cc7-1634-4ec1-86e2-28aeeb94a025"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:30:31.186726 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.186643 2541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-credential-socket\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:31.188587 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.188555 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "da531cc7-1634-4ec1-86e2-28aeeb94a025" (UID: "da531cc7-1634-4ec1-86e2-28aeeb94a025"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:31.188704 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.188682 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "da531cc7-1634-4ec1-86e2-28aeeb94a025" (UID: "da531cc7-1634-4ec1-86e2-28aeeb94a025"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:30:31.189132 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.189114 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da531cc7-1634-4ec1-86e2-28aeeb94a025-kube-api-access-xcwdd" (OuterVolumeSpecName: "kube-api-access-xcwdd") pod "da531cc7-1634-4ec1-86e2-28aeeb94a025" (UID: "da531cc7-1634-4ec1-86e2-28aeeb94a025"). InnerVolumeSpecName "kube-api-access-xcwdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:30:31.189179 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.189141 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-token" (OuterVolumeSpecName: "istio-token") pod "da531cc7-1634-4ec1-86e2-28aeeb94a025" (UID: "da531cc7-1634-4ec1-86e2-28aeeb94a025"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:30:31.287875 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.287777 2541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-token\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:31.287875 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.287812 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcwdd\" (UniqueName: \"kubernetes.io/projected/da531cc7-1634-4ec1-86e2-28aeeb94a025-kube-api-access-xcwdd\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:31.287875 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.287823 2541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-podinfo\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:31.287875 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.287831 2541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-envoy\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:31.287875 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.287839 2541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-workload-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:31.287875 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.287847 2541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-istio-data\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:31.287875 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.287854 2541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da531cc7-1634-4ec1-86e2-28aeeb94a025-istiod-ca-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:31.287875 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.287862 2541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da531cc7-1634-4ec1-86e2-28aeeb94a025-workload-socket\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:31.529067 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.529029 2541 generic.go:356] "Generic (PLEG): container finished" podID="da531cc7-1634-4ec1-86e2-28aeeb94a025" containerID="fae95858fc1456438959cd87f4c871965b5a59930f52d5204752c377f8fff3c1" exitCode=0 Sep 10 16:30:31.529282 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.529087 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" Sep 10 16:30:31.529282 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.529125 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" event={"ID":"da531cc7-1634-4ec1-86e2-28aeeb94a025","Type":"ContainerDied","Data":"fae95858fc1456438959cd87f4c871965b5a59930f52d5204752c377f8fff3c1"} Sep 10 16:30:31.529282 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.529196 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt" event={"ID":"da531cc7-1634-4ec1-86e2-28aeeb94a025","Type":"ContainerDied","Data":"ba1792c3f105d6aed764b9c278f7fe602b26e29dfad2727d2d4625c328e115a6"} Sep 10 16:30:31.529282 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.529230 2541 scope.go:117] "RemoveContainer" containerID="fae95858fc1456438959cd87f4c871965b5a59930f52d5204752c377f8fff3c1" Sep 10 16:30:31.540743 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.540718 2541 scope.go:117] "RemoveContainer" containerID="8dc8d63a705bbfe3214218d1bc5507fb0ce6b57250a4a15c10ee422612a86115" Sep 10 16:30:31.550803 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.550779 2541 scope.go:117] "RemoveContainer" containerID="fae95858fc1456438959cd87f4c871965b5a59930f52d5204752c377f8fff3c1" Sep 10 16:30:31.551204 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:30:31.551159 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fae95858fc1456438959cd87f4c871965b5a59930f52d5204752c377f8fff3c1\": container with ID starting with fae95858fc1456438959cd87f4c871965b5a59930f52d5204752c377f8fff3c1 not found: ID does not exist" containerID="fae95858fc1456438959cd87f4c871965b5a59930f52d5204752c377f8fff3c1" Sep 10 16:30:31.551286 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.551213 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fae95858fc1456438959cd87f4c871965b5a59930f52d5204752c377f8fff3c1"} err="failed to get container status \"fae95858fc1456438959cd87f4c871965b5a59930f52d5204752c377f8fff3c1\": rpc error: code = NotFound desc = could not find container \"fae95858fc1456438959cd87f4c871965b5a59930f52d5204752c377f8fff3c1\": container with ID starting with fae95858fc1456438959cd87f4c871965b5a59930f52d5204752c377f8fff3c1 not found: ID does not exist" Sep 10 16:30:31.551286 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.551237 2541 scope.go:117] "RemoveContainer" containerID="8dc8d63a705bbfe3214218d1bc5507fb0ce6b57250a4a15c10ee422612a86115" Sep 10 16:30:31.551861 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:30:31.551661 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc8d63a705bbfe3214218d1bc5507fb0ce6b57250a4a15c10ee422612a86115\": container with ID starting with 8dc8d63a705bbfe3214218d1bc5507fb0ce6b57250a4a15c10ee422612a86115 not found: ID does not exist" containerID="8dc8d63a705bbfe3214218d1bc5507fb0ce6b57250a4a15c10ee422612a86115" Sep 10 16:30:31.552059 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.552025 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc8d63a705bbfe3214218d1bc5507fb0ce6b57250a4a15c10ee422612a86115"} err="failed to get container status \"8dc8d63a705bbfe3214218d1bc5507fb0ce6b57250a4a15c10ee422612a86115\": rpc error: code = NotFound desc = could not find container \"8dc8d63a705bbfe3214218d1bc5507fb0ce6b57250a4a15c10ee422612a86115\": container with ID starting with 8dc8d63a705bbfe3214218d1bc5507fb0ce6b57250a4a15c10ee422612a86115 not found: ID does not exist" Sep 10 16:30:31.555075 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.555041 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt"] Sep 10 16:30:31.557913 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:31.557880 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ab677e-openshift-default-56d9b6cb5b-nwppt"] Sep 10 16:30:32.959061 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:32.959019 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:30:32.981441 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:32.981406 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da531cc7-1634-4ec1-86e2-28aeeb94a025" path="/var/lib/kubelet/pods/da531cc7-1634-4ec1-86e2-28aeeb94a025/volumes" Sep 10 16:30:34.286120 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:34.286084 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:36.228282 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:36.228245 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:38.195408 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.195361 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" podUID="0c85386e-6f75-4bfd-aa2a-5bd010307b3f" containerName="main" containerID="cri-o://c10cbbe9acb102fca75fee5ebecbbc6aadca2a8768d06397f04cc65511ff4320" gracePeriod=45 Sep 10 16:30:38.462172 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.462096 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:30:38.563942 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.563904 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glb8l\" (UniqueName: \"kubernetes.io/projected/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-kube-api-access-glb8l\") pod \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " Sep 10 16:30:38.564162 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.563977 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-tls-certs\") pod \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " Sep 10 16:30:38.564162 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.564012 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-dshm\") pod \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " Sep 10 16:30:38.564162 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.564061 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-home\") pod \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " Sep 10 16:30:38.564162 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.564080 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-model-cache\") pod \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " Sep 10 16:30:38.564162 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.564130 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-tmp-dir\") pod \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\" (UID: \"0c85386e-6f75-4bfd-aa2a-5bd010307b3f\") " Sep 10 16:30:38.564476 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.564377 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-home" (OuterVolumeSpecName: "home") pod "0c85386e-6f75-4bfd-aa2a-5bd010307b3f" (UID: "0c85386e-6f75-4bfd-aa2a-5bd010307b3f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:38.564476 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.564431 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c85386e-6f75-4bfd-aa2a-5bd010307b3f" (UID: "0c85386e-6f75-4bfd-aa2a-5bd010307b3f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:38.564558 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.564507 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c85386e-6f75-4bfd-aa2a-5bd010307b3f" (UID: "0c85386e-6f75-4bfd-aa2a-5bd010307b3f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:38.564558 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.564530 2541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-home\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:38.566518 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.566479 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-dshm" (OuterVolumeSpecName: "dshm") pod "0c85386e-6f75-4bfd-aa2a-5bd010307b3f" (UID: "0c85386e-6f75-4bfd-aa2a-5bd010307b3f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:38.566646 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.566518 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-kube-api-access-glb8l" (OuterVolumeSpecName: "kube-api-access-glb8l") pod "0c85386e-6f75-4bfd-aa2a-5bd010307b3f" (UID: "0c85386e-6f75-4bfd-aa2a-5bd010307b3f"). InnerVolumeSpecName "kube-api-access-glb8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:30:38.566646 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.566524 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c85386e-6f75-4bfd-aa2a-5bd010307b3f" (UID: "0c85386e-6f75-4bfd-aa2a-5bd010307b3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:30:38.579912 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.579879 2541 generic.go:356] "Generic (PLEG): container finished" podID="0c85386e-6f75-4bfd-aa2a-5bd010307b3f" containerID="c10cbbe9acb102fca75fee5ebecbbc6aadca2a8768d06397f04cc65511ff4320" exitCode=0 Sep 10 16:30:38.580105 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.579933 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" Sep 10 16:30:38.580105 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.579947 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" event={"ID":"0c85386e-6f75-4bfd-aa2a-5bd010307b3f","Type":"ContainerDied","Data":"c10cbbe9acb102fca75fee5ebecbbc6aadca2a8768d06397f04cc65511ff4320"} Sep 10 16:30:38.580105 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.579994 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs" event={"ID":"0c85386e-6f75-4bfd-aa2a-5bd010307b3f","Type":"ContainerDied","Data":"2c4a30f4f74313ddbcdd6effb964e8c2fed58b1f9eb6b967d7f98db248424a7c"} Sep 10 16:30:38.580105 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.580013 2541 scope.go:117] "RemoveContainer" containerID="c10cbbe9acb102fca75fee5ebecbbc6aadca2a8768d06397f04cc65511ff4320" Sep 10 16:30:38.592908 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.592886 2541 scope.go:117] "RemoveContainer" containerID="c10cbbe9acb102fca75fee5ebecbbc6aadca2a8768d06397f04cc65511ff4320" Sep 10 16:30:38.593257 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:30:38.593228 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c10cbbe9acb102fca75fee5ebecbbc6aadca2a8768d06397f04cc65511ff4320\": container with ID starting with c10cbbe9acb102fca75fee5ebecbbc6aadca2a8768d06397f04cc65511ff4320 not found: ID does not exist" containerID="c10cbbe9acb102fca75fee5ebecbbc6aadca2a8768d06397f04cc65511ff4320" Sep 10 16:30:38.593354 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.593267 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c10cbbe9acb102fca75fee5ebecbbc6aadca2a8768d06397f04cc65511ff4320"} err="failed to get container status \"c10cbbe9acb102fca75fee5ebecbbc6aadca2a8768d06397f04cc65511ff4320\": rpc error: code = NotFound desc = could not find container \"c10cbbe9acb102fca75fee5ebecbbc6aadca2a8768d06397f04cc65511ff4320\": container with ID starting with c10cbbe9acb102fca75fee5ebecbbc6aadca2a8768d06397f04cc65511ff4320 not found: ID does not exist" Sep 10 16:30:38.606226 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.606182 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs"] Sep 10 16:30:38.612206 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.612169 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b73303/rate-test-model-b73303-kserve-895747d75-g8ffs"] Sep 10 16:30:38.665579 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.665540 2541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-dshm\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:38.665754 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.665605 2541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-model-cache\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:38.665754 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.665615 2541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-tmp-dir\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:38.665754 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.665625 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glb8l\" (UniqueName: \"kubernetes.io/projected/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-kube-api-access-glb8l\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:38.665754 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.665636 2541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c85386e-6f75-4bfd-aa2a-5bd010307b3f-tls-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:38.981211 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:38.981177 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c85386e-6f75-4bfd-aa2a-5bd010307b3f" path="/var/lib/kubelet/pods/0c85386e-6f75-4bfd-aa2a-5bd010307b3f/volumes" Sep 10 16:30:41.273906 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:41.273864 2541 scope.go:117] "RemoveContainer" containerID="0604a4b03ec3c251c9a6b2c397d75fd7560f37ae1d24335ee742848ef358114b" Sep 10 16:30:41.897722 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:41.897682 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:41.907191 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:41.907152 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cde4a2-29817630-48fnl"] Sep 10 16:30:41.914420 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:41.914381 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cde4a2-29817630-48fnl"] Sep 10 16:30:42.007181 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:42.007142 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:42.100569 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:42.100538 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:42.174549 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:42.174412 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cde4a2-zjzkh"] Sep 10 16:30:42.177067 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:42.177028 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cde4a2-zjzkh"] Sep 10 16:30:42.179716 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:42.179671 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:42.254606 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:42.254570 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:42.563672 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:42.563570 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:42.980756 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:42.980703 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="465e85db-ee02-44e1-b063-2aabec5c113e" path="/var/lib/kubelet/pods/465e85db-ee02-44e1-b063-2aabec5c113e/volumes" Sep 10 16:30:42.981122 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:42.981101 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2f6ee1e-2027-44e5-ace6-cf30083ffcde" path="/var/lib/kubelet/pods/d2f6ee1e-2027-44e5-ace6-cf30083ffcde/volumes" Sep 10 16:30:48.269039 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:48.268993 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:30:49.683550 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:49.683513 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:49.785942 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:49.785900 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:49.947423 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:49.941620 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh"] Sep 10 16:30:49.947423 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:49.942009 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" podUID="0dcdaad8-c65f-4fc3-b952-1c4521feeae5" containerName="istio-proxy" containerID="cri-o://13eca48ade8b780096521480b72fcc73641e2b911a083cd46ffb0bd238e55719" gracePeriod=30 Sep 10 16:30:49.959592 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:49.959551 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:50.374454 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.374417 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29fd7d-nx5j5"] Sep 10 16:30:50.374845 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.374831 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c85386e-6f75-4bfd-aa2a-5bd010307b3f" containerName="main" Sep 10 16:30:50.374845 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.374845 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c85386e-6f75-4bfd-aa2a-5bd010307b3f" containerName="main" Sep 10 16:30:50.374957 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.374860 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da531cc7-1634-4ec1-86e2-28aeeb94a025" containerName="istio-proxy" Sep 10 16:30:50.374957 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.374866 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="da531cc7-1634-4ec1-86e2-28aeeb94a025" containerName="istio-proxy" Sep 10 16:30:50.374957 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.374875 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da531cc7-1634-4ec1-86e2-28aeeb94a025" containerName="istio-proxy" Sep 10 16:30:50.374957 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.374880 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="da531cc7-1634-4ec1-86e2-28aeeb94a025" containerName="istio-proxy" Sep 10 16:30:50.374957 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.374904 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="465e85db-ee02-44e1-b063-2aabec5c113e" containerName="revoke-keys" Sep 10 16:30:50.374957 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.374909 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="465e85db-ee02-44e1-b063-2aabec5c113e" containerName="revoke-keys" Sep 10 16:30:50.375145 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.374969 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c85386e-6f75-4bfd-aa2a-5bd010307b3f" containerName="main" Sep 10 16:30:50.375145 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.374979 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="da531cc7-1634-4ec1-86e2-28aeeb94a025" containerName="istio-proxy" Sep 10 16:30:50.375145 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.374987 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="da531cc7-1634-4ec1-86e2-28aeeb94a025" containerName="istio-proxy" Sep 10 16:30:50.375145 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.374992 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="465e85db-ee02-44e1-b063-2aabec5c113e" containerName="revoke-keys" Sep 10 16:30:50.378822 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.378797 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29fd7d-nx5j5" Sep 10 16:30:50.381636 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.381598 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l2dj2\"" Sep 10 16:30:50.381879 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.381655 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:30:50.390772 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.390738 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29fd7d-nx5j5"] Sep 10 16:30:50.482100 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.482062 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2eae8c-78db-4316-9a46-798c822db550-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-29fd7d-nx5j5\" (UID: \"7b2eae8c-78db-4316-9a46-798c822db550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29fd7d-nx5j5" Sep 10 16:30:50.509289 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.509251 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:50.583249 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.583212 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2eae8c-78db-4316-9a46-798c822db550-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-29fd7d-nx5j5\" (UID: \"7b2eae8c-78db-4316-9a46-798c822db550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29fd7d-nx5j5" Sep 10 16:30:50.583884 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.583864 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2eae8c-78db-4316-9a46-798c822db550-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-29fd7d-nx5j5\" (UID: \"7b2eae8c-78db-4316-9a46-798c822db550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29fd7d-nx5j5" Sep 10 16:30:50.693074 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.692965 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29fd7d-nx5j5" Sep 10 16:30:50.997011 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:50.996975 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29fd7d-nx5j5"] Sep 10 16:30:50.998192 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:30:50.998163 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2eae8c_78db_4316_9a46_798c822db550.slice/crio-45cefce881f5cdf20433c8636d8edd99ffc38bceed4b32700a7bc7b59ba5cd31 WatchSource:0}: Error finding container 45cefce881f5cdf20433c8636d8edd99ffc38bceed4b32700a7bc7b59ba5cd31: Status 404 returned error can't find the container with id 45cefce881f5cdf20433c8636d8edd99ffc38bceed4b32700a7bc7b59ba5cd31 Sep 10 16:30:51.415427 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:51.415393 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:51.669378 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:51.669335 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29fd7d-nx5j5" event={"ID":"7b2eae8c-78db-4316-9a46-798c822db550","Type":"ContainerStarted","Data":"a96bba79d30186e7840c95efee33c5d53b6163ce46803dee175cf5f1c60af81d"} Sep 10 16:30:51.669562 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:51.669403 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29fd7d-nx5j5" event={"ID":"7b2eae8c-78db-4316-9a46-798c822db550","Type":"ContainerStarted","Data":"45cefce881f5cdf20433c8636d8edd99ffc38bceed4b32700a7bc7b59ba5cd31"} Sep 10 16:30:51.707829 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:51.707777 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29fd7d-nx5j5" podStartSLOduration=1.707757709 podStartE2EDuration="1.707757709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:30:51.706154681 +0000 UTC m=+1751.229399878" watchObservedRunningTime="2026-09-10 16:30:51.707757709 +0000 UTC m=+1751.231002905" Sep 10 16:30:52.299318 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:52.299269 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:52.678004 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:52.677967 2541 generic.go:356] "Generic (PLEG): container finished" podID="7b2eae8c-78db-4316-9a46-798c822db550" containerID="a96bba79d30186e7840c95efee33c5d53b6163ce46803dee175cf5f1c60af81d" exitCode=0 Sep 10 16:30:52.678223 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:52.678059 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29fd7d-nx5j5" event={"ID":"7b2eae8c-78db-4316-9a46-798c822db550","Type":"ContainerDied","Data":"a96bba79d30186e7840c95efee33c5d53b6163ce46803dee175cf5f1c60af81d"} Sep 10 16:30:53.832136 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:53.832095 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29fd7d-nx5j5" Sep 10 16:30:53.916070 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:53.916030 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2eae8c-78db-4316-9a46-798c822db550-maas-api-service-ca\") pod \"7b2eae8c-78db-4316-9a46-798c822db550\" (UID: \"7b2eae8c-78db-4316-9a46-798c822db550\") " Sep 10 16:30:53.916477 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:53.916452 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b2eae8c-78db-4316-9a46-798c822db550-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7b2eae8c-78db-4316-9a46-798c822db550" (UID: "7b2eae8c-78db-4316-9a46-798c822db550"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:30:54.017075 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:54.016975 2541 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2eae8c-78db-4316-9a46-798c822db550-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:54.693030 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:54.692999 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29fd7d-nx5j5" Sep 10 16:30:54.693228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:54.693031 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29fd7d-nx5j5" event={"ID":"7b2eae8c-78db-4316-9a46-798c822db550","Type":"ContainerDied","Data":"45cefce881f5cdf20433c8636d8edd99ffc38bceed4b32700a7bc7b59ba5cd31"} Sep 10 16:30:54.693228 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:54.693115 2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45cefce881f5cdf20433c8636d8edd99ffc38bceed4b32700a7bc7b59ba5cd31" Sep 10 16:30:55.214628 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.214547 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:30:55.329427 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.329383 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-data\") pod \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " Sep 10 16:30:55.329624 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.329445 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-token\") pod \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " Sep 10 16:30:55.329624 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.329481 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istiod-ca-cert\") pod \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " Sep 10 16:30:55.329624 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.329533 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-envoy\") pod \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " Sep 10 16:30:55.329624 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.329618 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-podinfo\") pod \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " Sep 10 16:30:55.329898 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.329643 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfxwk\" (UniqueName: \"kubernetes.io/projected/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-kube-api-access-nfxwk\") pod \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " Sep 10 16:30:55.329898 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.329671 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-workload-socket\") pod \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " Sep 10 16:30:55.329898 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.329705 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-workload-certs\") pod \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " Sep 10 16:30:55.329898 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.329766 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-credential-socket\") pod \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\" (UID: \"0dcdaad8-c65f-4fc3-b952-1c4521feeae5\") " Sep 10 16:30:55.330125 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.330019 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0dcdaad8-c65f-4fc3-b952-1c4521feeae5" (UID: "0dcdaad8-c65f-4fc3-b952-1c4521feeae5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:55.330125 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.330044 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0dcdaad8-c65f-4fc3-b952-1c4521feeae5" (UID: "0dcdaad8-c65f-4fc3-b952-1c4521feeae5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:30:55.330239 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.330139 2541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istiod-ca-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:55.330239 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.330157 2541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-workload-socket\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:55.330398 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.330236 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0dcdaad8-c65f-4fc3-b952-1c4521feeae5" (UID: "0dcdaad8-c65f-4fc3-b952-1c4521feeae5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:55.330398 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.330249 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0dcdaad8-c65f-4fc3-b952-1c4521feeae5" (UID: "0dcdaad8-c65f-4fc3-b952-1c4521feeae5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:55.330398 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.330312 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-data" (OuterVolumeSpecName: "istio-data") pod "0dcdaad8-c65f-4fc3-b952-1c4521feeae5" (UID: "0dcdaad8-c65f-4fc3-b952-1c4521feeae5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:55.332178 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.332146 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-token" (OuterVolumeSpecName: "istio-token") pod "0dcdaad8-c65f-4fc3-b952-1c4521feeae5" (UID: "0dcdaad8-c65f-4fc3-b952-1c4521feeae5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:30:55.332448 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.332428 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0dcdaad8-c65f-4fc3-b952-1c4521feeae5" (UID: "0dcdaad8-c65f-4fc3-b952-1c4521feeae5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:55.332645 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.332615 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0dcdaad8-c65f-4fc3-b952-1c4521feeae5" (UID: "0dcdaad8-c65f-4fc3-b952-1c4521feeae5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:30:55.332724 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.332618 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-kube-api-access-nfxwk" (OuterVolumeSpecName: "kube-api-access-nfxwk") pod "0dcdaad8-c65f-4fc3-b952-1c4521feeae5" (UID: "0dcdaad8-c65f-4fc3-b952-1c4521feeae5"). InnerVolumeSpecName "kube-api-access-nfxwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:30:55.430885 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.430842 2541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-workload-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:55.430885 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.430886 2541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-credential-socket\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:55.431139 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.430903 2541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-data\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:55.431139 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.430916 2541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-token\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:55.431139 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.430928 2541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-envoy\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:55.431139 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.430940 2541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-istio-podinfo\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:55.431139 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.430952 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfxwk\" (UniqueName: \"kubernetes.io/projected/0dcdaad8-c65f-4fc3-b952-1c4521feeae5-kube-api-access-nfxwk\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:30:55.701419 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.701383 2541 generic.go:356] "Generic (PLEG): container finished" podID="0dcdaad8-c65f-4fc3-b952-1c4521feeae5" containerID="13eca48ade8b780096521480b72fcc73641e2b911a083cd46ffb0bd238e55719" exitCode=0 Sep 10 16:30:55.701419 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.701425 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" event={"ID":"0dcdaad8-c65f-4fc3-b952-1c4521feeae5","Type":"ContainerDied","Data":"13eca48ade8b780096521480b72fcc73641e2b911a083cd46ffb0bd238e55719"} Sep 10 16:30:55.701665 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.701439 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" Sep 10 16:30:55.701665 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.701467 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh" event={"ID":"0dcdaad8-c65f-4fc3-b952-1c4521feeae5","Type":"ContainerDied","Data":"fcbaa00c396f507e9516c50c86d88cf7553e19f2ceefe2d4caf255b2608e35d5"} Sep 10 16:30:55.701665 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.701492 2541 scope.go:117] "RemoveContainer" containerID="13eca48ade8b780096521480b72fcc73641e2b911a083cd46ffb0bd238e55719" Sep 10 16:30:55.713391 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.713370 2541 scope.go:117] "RemoveContainer" containerID="a27820173fe7ec34d9f9900a4aad4f3095a530cc823356754b83c7a0ddde0978" Sep 10 16:30:55.723592 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.723561 2541 scope.go:117] "RemoveContainer" containerID="13eca48ade8b780096521480b72fcc73641e2b911a083cd46ffb0bd238e55719" Sep 10 16:30:55.724370 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:30:55.724273 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13eca48ade8b780096521480b72fcc73641e2b911a083cd46ffb0bd238e55719\": container with ID starting with 13eca48ade8b780096521480b72fcc73641e2b911a083cd46ffb0bd238e55719 not found: ID does not exist" containerID="13eca48ade8b780096521480b72fcc73641e2b911a083cd46ffb0bd238e55719" Sep 10 16:30:55.724500 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.724383 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13eca48ade8b780096521480b72fcc73641e2b911a083cd46ffb0bd238e55719"} err="failed to get container status \"13eca48ade8b780096521480b72fcc73641e2b911a083cd46ffb0bd238e55719\": rpc error: code = NotFound desc = could not find container \"13eca48ade8b780096521480b72fcc73641e2b911a083cd46ffb0bd238e55719\": container with ID starting with 13eca48ade8b780096521480b72fcc73641e2b911a083cd46ffb0bd238e55719 not found: ID does not exist" Sep 10 16:30:55.724500 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.724414 2541 scope.go:117] "RemoveContainer" containerID="a27820173fe7ec34d9f9900a4aad4f3095a530cc823356754b83c7a0ddde0978" Sep 10 16:30:55.724780 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:30:55.724757 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a27820173fe7ec34d9f9900a4aad4f3095a530cc823356754b83c7a0ddde0978\": container with ID starting with a27820173fe7ec34d9f9900a4aad4f3095a530cc823356754b83c7a0ddde0978 not found: ID does not exist" containerID="a27820173fe7ec34d9f9900a4aad4f3095a530cc823356754b83c7a0ddde0978" Sep 10 16:30:55.724832 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.724788 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27820173fe7ec34d9f9900a4aad4f3095a530cc823356754b83c7a0ddde0978"} err="failed to get container status \"a27820173fe7ec34d9f9900a4aad4f3095a530cc823356754b83c7a0ddde0978\": rpc error: code = NotFound desc = could not find container \"a27820173fe7ec34d9f9900a4aad4f3095a530cc823356754b83c7a0ddde0978\": container with ID starting with a27820173fe7ec34d9f9900a4aad4f3095a530cc823356754b83c7a0ddde0978 not found: ID does not exist" Sep 10 16:30:55.727188 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.727153 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh"] Sep 10 16:30:55.731757 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:55.731723 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-cde4a2-openshift-default-df8984f44-kcqgh"] Sep 10 16:30:56.750833 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:56.750799 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:56.980239 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:56.980195 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dcdaad8-c65f-4fc3-b952-1c4521feeae5" path="/var/lib/kubelet/pods/0dcdaad8-c65f-4fc3-b952-1c4521feeae5/volumes" Sep 10 16:30:57.478373 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.478331 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:57.770109 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.770020 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk"] Sep 10 16:30:57.770529 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.770513 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dcdaad8-c65f-4fc3-b952-1c4521feeae5" containerName="istio-proxy" Sep 10 16:30:57.770580 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.770531 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dcdaad8-c65f-4fc3-b952-1c4521feeae5" containerName="istio-proxy" Sep 10 16:30:57.770580 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.770546 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dcdaad8-c65f-4fc3-b952-1c4521feeae5" containerName="istio-proxy" Sep 10 16:30:57.770580 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.770551 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dcdaad8-c65f-4fc3-b952-1c4521feeae5" containerName="istio-proxy" Sep 10 16:30:57.770580 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.770560 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b2eae8c-78db-4316-9a46-798c822db550" containerName="revoke-keys" Sep 10 16:30:57.770580 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.770566 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b2eae8c-78db-4316-9a46-798c822db550" containerName="revoke-keys" Sep 10 16:30:57.770736 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.770630 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b2eae8c-78db-4316-9a46-798c822db550" containerName="revoke-keys" Sep 10 16:30:57.770736 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.770638 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dcdaad8-c65f-4fc3-b952-1c4521feeae5" containerName="istio-proxy" Sep 10 16:30:57.770736 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.770643 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dcdaad8-c65f-4fc3-b952-1c4521feeae5" containerName="istio-proxy" Sep 10 16:30:57.775438 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.775410 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.777625 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.777600 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-275a06-openshift-default-dockercfg-sw5v7\"" Sep 10 16:30:57.786485 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.786449 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk"] Sep 10 16:30:57.799356 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.799291 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:57.856779 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.856742 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-podinfo\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.857099 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.857076 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-envoy\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.857266 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.857248 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xqqt\" (UniqueName: \"kubernetes.io/projected/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-kube-api-access-5xqqt\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.857478 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.857458 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-token\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.857609 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.857585 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istiod-ca-cert\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.857712 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.857694 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-credential-socket\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.857807 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.857779 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-workload-certs\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.858086 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.858043 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-data\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.858558 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.858522 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-workload-socket\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.959650 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.959609 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-data\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.959650 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.959658 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-workload-socket\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.959924 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.959688 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-podinfo\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.959924 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.959705 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-envoy\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.959924 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.959722 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xqqt\" (UniqueName: \"kubernetes.io/projected/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-kube-api-access-5xqqt\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.959924 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.959738 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-token\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.959924 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.959755 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istiod-ca-cert\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.959924 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.959817 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-credential-socket\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.959924 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.959878 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-workload-certs\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.960344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.960081 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-data\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.960344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.960146 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-workload-socket\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.960344 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.960212 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-workload-certs\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.960522 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.960382 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-credential-socket\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.960664 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.960642 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istiod-ca-cert\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.962407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.962381 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-envoy\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.962610 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.962593 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-podinfo\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.971055 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.971018 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-token\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:57.971527 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:57.971505 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xqqt\" (UniqueName: \"kubernetes.io/projected/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-kube-api-access-5xqqt\") pod \"e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:58.089939 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:58.089846 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:58.380553 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:58.380520 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk"] Sep 10 16:30:58.382359 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:30:58.382293 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f0a5c3_cd3a_4186_a3fd_df5ca7b08060.slice/crio-071ccab4f87db1d81f3be4fbd892e6e91bd1de42d8c766c180eaf0060caa8c54 WatchSource:0}: Error finding container 071ccab4f87db1d81f3be4fbd892e6e91bd1de42d8c766c180eaf0060caa8c54: Status 404 returned error can't find the container with id 071ccab4f87db1d81f3be4fbd892e6e91bd1de42d8c766c180eaf0060caa8c54 Sep 10 16:30:58.384935 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:58.384900 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:30:58.385049 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:58.384973 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:30:58.385049 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:58.385002 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:30:58.394218 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:58.394179 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:58.732208 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:58.732110 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" event={"ID":"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060","Type":"ContainerStarted","Data":"2266be7fb8a9ff53af77dbf806db6e4c29f74d48c4b60b38cc89d486c57f9e55"} Sep 10 16:30:58.732208 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:58.732160 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" event={"ID":"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060","Type":"ContainerStarted","Data":"071ccab4f87db1d81f3be4fbd892e6e91bd1de42d8c766c180eaf0060caa8c54"} Sep 10 16:30:58.756544 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:58.756467 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" podStartSLOduration=1.75644756 podStartE2EDuration="1.75644756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:30:58.753670706 +0000 UTC m=+1758.276915902" watchObservedRunningTime="2026-09-10 16:30:58.75644756 +0000 UTC m=+1758.279692757" Sep 10 16:30:59.090945 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:59.090851 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:59.097068 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:59.097039 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:59.739193 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:59.739159 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:30:59.740402 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:30:59.740374 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:31:00.525799 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:00.525752 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:00.812740 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:00.812645 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:00.911767 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:00.911707 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:01.417589 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:01.417543 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:01.514637 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:01.514587 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:01.617540 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:01.617488 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:01.682880 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:01.682780 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29fd7d-nx5j5"] Sep 10 16:31:01.687602 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:01.687565 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-29fd7d-nx5j5"] Sep 10 16:31:01.699179 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:01.699126 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:01.947017 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:01.946911 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:02.048674 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:02.048641 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:02.124418 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:02.124384 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:02.452839 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:02.452793 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:02.661967 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:02.661931 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:02.761337 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:02.761221 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:02.859394 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:02.859340 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc"] Sep 10 16:31:02.863614 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:02.863581 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc" Sep 10 16:31:02.865856 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:02.865827 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-275a06\"" Sep 10 16:31:02.866025 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:02.865883 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-275a06-dockercfg-gkwbf\"" Sep 10 16:31:02.873443 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:02.873405 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc"] Sep 10 16:31:02.910925 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:02.910872 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad02186-02e9-4565-9c18-34a59d63a36e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc\" (UID: \"bad02186-02e9-4565-9c18-34a59d63a36e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc" Sep 10 16:31:02.911143 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:02.910945 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hgs5\" (UniqueName: \"kubernetes.io/projected/bad02186-02e9-4565-9c18-34a59d63a36e-kube-api-access-4hgs5\") pod \"payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc\" (UID: \"bad02186-02e9-4565-9c18-34a59d63a36e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc" Sep 10 16:31:02.980404 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:02.980370 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b2eae8c-78db-4316-9a46-798c822db550" path="/var/lib/kubelet/pods/7b2eae8c-78db-4316-9a46-798c822db550/volumes" Sep 10 16:31:03.012376 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:03.012273 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad02186-02e9-4565-9c18-34a59d63a36e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc\" (UID: \"bad02186-02e9-4565-9c18-34a59d63a36e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc" Sep 10 16:31:03.012546 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:03.012394 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hgs5\" (UniqueName: \"kubernetes.io/projected/bad02186-02e9-4565-9c18-34a59d63a36e-kube-api-access-4hgs5\") pod \"payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc\" (UID: \"bad02186-02e9-4565-9c18-34a59d63a36e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc" Sep 10 16:31:03.013118 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:03.013094 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad02186-02e9-4565-9c18-34a59d63a36e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc\" (UID: \"bad02186-02e9-4565-9c18-34a59d63a36e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc" Sep 10 16:31:03.023582 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:03.023545 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hgs5\" (UniqueName: \"kubernetes.io/projected/bad02186-02e9-4565-9c18-34a59d63a36e-kube-api-access-4hgs5\") pod \"payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc\" (UID: \"bad02186-02e9-4565-9c18-34a59d63a36e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc" Sep 10 16:31:03.178154 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:03.178111 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc" Sep 10 16:31:03.458022 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:03.457990 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc"] Sep 10 16:31:03.459992 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:31:03.459964 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad02186_02e9_4565_9c18_34a59d63a36e.slice/crio-43f8d280e9d5ba6ba4d268ff245d5999cd8696a85adc3b8a4c2f0e69e3865335 WatchSource:0}: Error finding container 43f8d280e9d5ba6ba4d268ff245d5999cd8696a85adc3b8a4c2f0e69e3865335: Status 404 returned error can't find the container with id 43f8d280e9d5ba6ba4d268ff245d5999cd8696a85adc3b8a4c2f0e69e3865335 Sep 10 16:31:03.770557 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:03.770457 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc" event={"ID":"bad02186-02e9-4565-9c18-34a59d63a36e","Type":"ContainerStarted","Data":"46de7140380a5b0c4e7d0869b246c83c011c0d573db091039c6c3fd5b259770c"} Sep 10 16:31:03.770952 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:03.770604 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc" Sep 10 16:31:03.770952 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:03.770648 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc" event={"ID":"bad02186-02e9-4565-9c18-34a59d63a36e","Type":"ContainerStarted","Data":"43f8d280e9d5ba6ba4d268ff245d5999cd8696a85adc3b8a4c2f0e69e3865335"} Sep 10 16:31:03.791541 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:03.791435 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc" podStartSLOduration=1.791412742 podStartE2EDuration="1.791412742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:31:03.790549931 +0000 UTC m=+1763.313795128" watchObservedRunningTime="2026-09-10 16:31:03.791412742 +0000 UTC m=+1763.314657943" Sep 10 16:31:05.058392 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:05.058349 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:31:08.609671 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:08.609633 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:08.679180 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:08.679141 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:08.868957 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:08.868865 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:09.198603 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:09.198565 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90bd7e-9fxsj"] Sep 10 16:31:09.202573 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:09.202550 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90bd7e-9fxsj" Sep 10 16:31:09.204959 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:09.204923 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:31:09.205115 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:09.205074 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l2dj2\"" Sep 10 16:31:09.211731 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:09.211491 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90bd7e-9fxsj"] Sep 10 16:31:09.282631 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:09.282588 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4a1a29-be5b-4320-8de2-818b6535533d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-90bd7e-9fxsj\" (UID: \"5d4a1a29-be5b-4320-8de2-818b6535533d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90bd7e-9fxsj" Sep 10 16:31:09.307065 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:09.307022 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:09.383990 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:09.383938 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4a1a29-be5b-4320-8de2-818b6535533d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-90bd7e-9fxsj\" (UID: \"5d4a1a29-be5b-4320-8de2-818b6535533d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90bd7e-9fxsj" Sep 10 16:31:09.384702 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:09.384680 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4a1a29-be5b-4320-8de2-818b6535533d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-90bd7e-9fxsj\" (UID: \"5d4a1a29-be5b-4320-8de2-818b6535533d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90bd7e-9fxsj" Sep 10 16:31:09.515768 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:09.515659 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90bd7e-9fxsj" Sep 10 16:31:09.802652 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:09.802618 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90bd7e-9fxsj"] Sep 10 16:31:09.804458 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:31:09.804428 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d4a1a29_be5b_4320_8de2_818b6535533d.slice/crio-c036cef9a77d6da27cd0c606154ed8d7e9729068a7c05f239fe636a14e827298 WatchSource:0}: Error finding container c036cef9a77d6da27cd0c606154ed8d7e9729068a7c05f239fe636a14e827298: Status 404 returned error can't find the container with id c036cef9a77d6da27cd0c606154ed8d7e9729068a7c05f239fe636a14e827298 Sep 10 16:31:09.813642 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:09.813611 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90bd7e-9fxsj" event={"ID":"5d4a1a29-be5b-4320-8de2-818b6535533d","Type":"ContainerStarted","Data":"c036cef9a77d6da27cd0c606154ed8d7e9729068a7c05f239fe636a14e827298"} Sep 10 16:31:10.821186 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:10.821149 2541 generic.go:356] "Generic (PLEG): container finished" podID="5d4a1a29-be5b-4320-8de2-818b6535533d" containerID="a3064bd9b9151398c7563960e28068e28aa56097fa2893068741697b445b343f" exitCode=0 Sep 10 16:31:10.821628 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:10.821182 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90bd7e-9fxsj" event={"ID":"5d4a1a29-be5b-4320-8de2-818b6535533d","Type":"ContainerDied","Data":"a3064bd9b9151398c7563960e28068e28aa56097fa2893068741697b445b343f"} Sep 10 16:31:11.977612 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:11.977578 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90bd7e-9fxsj" Sep 10 16:31:12.110453 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:12.110346 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4a1a29-be5b-4320-8de2-818b6535533d-maas-api-service-ca\") pod \"5d4a1a29-be5b-4320-8de2-818b6535533d\" (UID: \"5d4a1a29-be5b-4320-8de2-818b6535533d\") " Sep 10 16:31:12.110766 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:12.110741 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d4a1a29-be5b-4320-8de2-818b6535533d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5d4a1a29-be5b-4320-8de2-818b6535533d" (UID: "5d4a1a29-be5b-4320-8de2-818b6535533d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:31:12.218587 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:12.212784 2541 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4a1a29-be5b-4320-8de2-818b6535533d-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:31:12.836923 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:12.836891 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90bd7e-9fxsj" Sep 10 16:31:12.837115 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:12.836926 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90bd7e-9fxsj" event={"ID":"5d4a1a29-be5b-4320-8de2-818b6535533d","Type":"ContainerDied","Data":"c036cef9a77d6da27cd0c606154ed8d7e9729068a7c05f239fe636a14e827298"} Sep 10 16:31:12.837115 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:12.836997 2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c036cef9a77d6da27cd0c606154ed8d7e9729068a7c05f239fe636a14e827298" Sep 10 16:31:14.781500 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:14.781449 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc" Sep 10 16:31:19.736804 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:19.736772 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:19.776836 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:19.776800 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g"] Sep 10 16:31:19.777089 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:19.777059 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g" podUID="5fbc7294-dcdd-4246-8142-0db96e7f6ab8" containerName="payload-pre-processing" containerID="cri-o://7601f9c5f7e1fba41c48e568c18576eb6a16731d1103424d63fe83e29a2ca250" gracePeriod=30 Sep 10 16:31:20.010664 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:20.010582 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90bd7e-9fxsj"] Sep 10 16:31:20.014293 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:20.014242 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90bd7e-9fxsj"] Sep 10 16:31:20.077674 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:20.077627 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g" Sep 10 16:31:20.196727 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:20.196618 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fbc7294-dcdd-4246-8142-0db96e7f6ab8-plugins-config-volume\") pod \"5fbc7294-dcdd-4246-8142-0db96e7f6ab8\" (UID: \"5fbc7294-dcdd-4246-8142-0db96e7f6ab8\") " Sep 10 16:31:20.196727 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:20.196694 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6x76\" (UniqueName: \"kubernetes.io/projected/5fbc7294-dcdd-4246-8142-0db96e7f6ab8-kube-api-access-d6x76\") pod \"5fbc7294-dcdd-4246-8142-0db96e7f6ab8\" (UID: \"5fbc7294-dcdd-4246-8142-0db96e7f6ab8\") " Sep 10 16:31:20.197142 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:20.197109 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fbc7294-dcdd-4246-8142-0db96e7f6ab8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5fbc7294-dcdd-4246-8142-0db96e7f6ab8" (UID: "5fbc7294-dcdd-4246-8142-0db96e7f6ab8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:31:20.199019 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:20.198988 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fbc7294-dcdd-4246-8142-0db96e7f6ab8-kube-api-access-d6x76" (OuterVolumeSpecName: "kube-api-access-d6x76") pod "5fbc7294-dcdd-4246-8142-0db96e7f6ab8" (UID: "5fbc7294-dcdd-4246-8142-0db96e7f6ab8"). InnerVolumeSpecName "kube-api-access-d6x76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:31:20.297835 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:20.297771 2541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fbc7294-dcdd-4246-8142-0db96e7f6ab8-plugins-config-volume\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:31:20.297835 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:20.297833 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6x76\" (UniqueName: \"kubernetes.io/projected/5fbc7294-dcdd-4246-8142-0db96e7f6ab8-kube-api-access-d6x76\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:31:20.890749 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:20.890705 2541 generic.go:356] "Generic (PLEG): container finished" podID="5fbc7294-dcdd-4246-8142-0db96e7f6ab8" containerID="7601f9c5f7e1fba41c48e568c18576eb6a16731d1103424d63fe83e29a2ca250" exitCode=0 Sep 10 16:31:20.891264 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:20.890758 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g" Sep 10 16:31:20.891264 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:20.890793 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g" event={"ID":"5fbc7294-dcdd-4246-8142-0db96e7f6ab8","Type":"ContainerDied","Data":"7601f9c5f7e1fba41c48e568c18576eb6a16731d1103424d63fe83e29a2ca250"} Sep 10 16:31:20.891264 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:20.890865 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g" event={"ID":"5fbc7294-dcdd-4246-8142-0db96e7f6ab8","Type":"ContainerDied","Data":"ca7000514671a755cd5eada1437e5df66d115d64582582f7aa833848eac055e6"} Sep 10 16:31:20.891264 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:20.890901 2541 scope.go:117] "RemoveContainer" containerID="7601f9c5f7e1fba41c48e568c18576eb6a16731d1103424d63fe83e29a2ca250" Sep 10 16:31:20.902709 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:20.902669 2541 scope.go:117] "RemoveContainer" containerID="7601f9c5f7e1fba41c48e568c18576eb6a16731d1103424d63fe83e29a2ca250" Sep 10 16:31:20.903242 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:31:20.903136 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7601f9c5f7e1fba41c48e568c18576eb6a16731d1103424d63fe83e29a2ca250\": container with ID starting with 7601f9c5f7e1fba41c48e568c18576eb6a16731d1103424d63fe83e29a2ca250 not found: ID does not exist" containerID="7601f9c5f7e1fba41c48e568c18576eb6a16731d1103424d63fe83e29a2ca250" Sep 10 16:31:20.903242 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:20.903180 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7601f9c5f7e1fba41c48e568c18576eb6a16731d1103424d63fe83e29a2ca250"} err="failed to get container status \"7601f9c5f7e1fba41c48e568c18576eb6a16731d1103424d63fe83e29a2ca250\": rpc error: code = NotFound desc = could not find container \"7601f9c5f7e1fba41c48e568c18576eb6a16731d1103424d63fe83e29a2ca250\": container with ID starting with 7601f9c5f7e1fba41c48e568c18576eb6a16731d1103424d63fe83e29a2ca250 not found: ID does not exist" Sep 10 16:31:20.916254 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:20.916215 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g"] Sep 10 16:31:20.920848 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:20.920804 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90bd7e-b4c88455d-r277g"] Sep 10 16:31:20.983213 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:20.983171 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d4a1a29-be5b-4320-8de2-818b6535533d" path="/var/lib/kubelet/pods/5d4a1a29-be5b-4320-8de2-818b6535533d/volumes" Sep 10 16:31:20.983712 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:20.983693 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fbc7294-dcdd-4246-8142-0db96e7f6ab8" path="/var/lib/kubelet/pods/5fbc7294-dcdd-4246-8142-0db96e7f6ab8/volumes" Sep 10 16:31:21.061242 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:21.061203 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:31:21.727459 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:21.727423 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:23.916074 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:23.916037 2541 generic.go:356] "Generic (PLEG): container finished" podID="00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" containerID="2266be7fb8a9ff53af77dbf806db6e4c29f74d48c4b60b38cc89d486c57f9e55" exitCode=0 Sep 10 16:31:23.916612 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:23.916095 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" event={"ID":"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060","Type":"ContainerDied","Data":"2266be7fb8a9ff53af77dbf806db6e4c29f74d48c4b60b38cc89d486c57f9e55"} Sep 10 16:31:23.916612 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:23.916595 2541 scope.go:117] "RemoveContainer" containerID="2266be7fb8a9ff53af77dbf806db6e4c29f74d48c4b60b38cc89d486c57f9e55" Sep 10 16:31:23.917884 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:23.917847 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:31:23.917998 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:23.917943 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:31:23.917998 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:23.917990 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:31:24.539525 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:24.539463 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:24.925506 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:24.925456 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" event={"ID":"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060","Type":"ContainerStarted","Data":"6a3b6951f686f4ee14d636396c0909146e01f613fa2c79fa87f10620747a31c3"} Sep 10 16:31:25.090748 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:25.090707 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:31:25.092244 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:25.092185 2541 patch_prober.go:28] interesting pod/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 10 16:31:25.092479 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:25.092289 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" podUID="00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 10 16:31:25.786075 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:25.786035 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:26.090859 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:26.090747 2541 patch_prober.go:28] interesting pod/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 10 16:31:26.090859 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:26.090828 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" podUID="00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 10 16:31:27.090737 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:27.090696 2541 patch_prober.go:28] interesting pod/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 10 16:31:27.090986 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:27.090768 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" podUID="00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 10 16:31:27.237236 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:27.237200 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:27.602370 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:27.602327 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:27.722049 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:27.722016 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:27.835463 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:27.835426 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4"] Sep 10 16:31:27.835664 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:27.835642 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" podUID="aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" containerName="istio-proxy" containerID="cri-o://3c83fe730036df781436136721e2dcd8f08c540e4470d255eeedd5f300f2d221" gracePeriod=30 Sep 10 16:31:27.874890 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:27.874846 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:28.091369 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:28.091293 2541 patch_prober.go:28] interesting pod/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 10 16:31:28.091997 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:28.091963 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" podUID="00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 10 16:31:29.090855 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:29.090813 2541 patch_prober.go:28] interesting pod/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 10 16:31:29.091174 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:29.091146 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" podUID="00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 10 16:31:29.739162 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:29.739107 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:31:30.090456 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:30.090350 2541 patch_prober.go:28] interesting pod/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 10 16:31:30.090456 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:30.090427 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" podUID="00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 10 16:31:30.973740 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:30.973698 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-85d2w_c9c7e7f6-7f76-493c-aa7a-f69a70592836/payload-processing/0.log" Sep 10 16:31:31.095611 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:31.095566 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:31:31.097000 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:31.096962 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:31:32.989658 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:32.989622 2541 generic.go:356] "Generic (PLEG): container finished" podID="aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" containerID="3c83fe730036df781436136721e2dcd8f08c540e4470d255eeedd5f300f2d221" exitCode=0 Sep 10 16:31:32.990173 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:32.989704 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" event={"ID":"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092","Type":"ContainerDied","Data":"3c83fe730036df781436136721e2dcd8f08c540e4470d255eeedd5f300f2d221"} Sep 10 16:31:32.990173 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:32.989779 2541 scope.go:117] "RemoveContainer" containerID="5a2ade10d48a325ea24de182bf0f085cb32b2d0ef8f5a29ae07e789a652dae6c" Sep 10 16:31:33.112217 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.112136 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:31:33.222295 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.222251 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-data\") pod \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " Sep 10 16:31:33.222515 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.222410 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-token\") pod \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " Sep 10 16:31:33.222515 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.222430 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istiod-ca-cert\") pod \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " Sep 10 16:31:33.222515 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.222452 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-envoy\") pod \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " Sep 10 16:31:33.222515 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.222474 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-workload-socket\") pod \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " Sep 10 16:31:33.222515 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.222499 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-workload-certs\") pod \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " Sep 10 16:31:33.222766 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.222573 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-data" (OuterVolumeSpecName: "istio-data") pod "aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" (UID: "aeef5b3d-8ba8-4901-a1d8-9135ec7b5092"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:31:33.222766 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.222597 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9qzx\" (UniqueName: \"kubernetes.io/projected/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-kube-api-access-k9qzx\") pod \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " Sep 10 16:31:33.222766 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.222665 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-podinfo\") pod \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " Sep 10 16:31:33.222766 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.222695 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-credential-socket\") pod \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\" (UID: \"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092\") " Sep 10 16:31:33.222970 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.222799 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" (UID: "aeef5b3d-8ba8-4901-a1d8-9135ec7b5092"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:31:33.222970 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.222925 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" (UID: "aeef5b3d-8ba8-4901-a1d8-9135ec7b5092"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:31:33.223074 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.223025 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" (UID: "aeef5b3d-8ba8-4901-a1d8-9135ec7b5092"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:31:33.223133 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.223088 2541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istiod-ca-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:31:33.223133 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.223097 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" (UID: "aeef5b3d-8ba8-4901-a1d8-9135ec7b5092"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:31:33.223133 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.223105 2541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-workload-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:31:33.223133 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.223133 2541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-data\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:31:33.225200 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.225152 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-kube-api-access-k9qzx" (OuterVolumeSpecName: "kube-api-access-k9qzx") pod "aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" (UID: "aeef5b3d-8ba8-4901-a1d8-9135ec7b5092"). InnerVolumeSpecName "kube-api-access-k9qzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:31:33.225376 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.225209 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-token" (OuterVolumeSpecName: "istio-token") pod "aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" (UID: "aeef5b3d-8ba8-4901-a1d8-9135ec7b5092"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:31:33.225726 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.225690 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" (UID: "aeef5b3d-8ba8-4901-a1d8-9135ec7b5092"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:31:33.225726 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.225694 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" (UID: "aeef5b3d-8ba8-4901-a1d8-9135ec7b5092"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:31:33.324091 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.324043 2541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-podinfo\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:31:33.324091 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.324082 2541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-credential-socket\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:31:33.324091 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.324093 2541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-token\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:31:33.324091 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.324102 2541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-istio-envoy\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:31:33.324452 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.324110 2541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-workload-socket\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:31:33.324452 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.324119 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9qzx\" (UniqueName: \"kubernetes.io/projected/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092-kube-api-access-k9qzx\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:31:33.957090 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:33.957045 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:31:34.000087 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:34.000055 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" Sep 10 16:31:34.000575 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:34.000090 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4" event={"ID":"aeef5b3d-8ba8-4901-a1d8-9135ec7b5092","Type":"ContainerDied","Data":"9a9477147287acd7d530da32d8b9268ec353728d0dafc2550288eb6f64df868c"} Sep 10 16:31:34.000575 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:34.000160 2541 scope.go:117] "RemoveContainer" containerID="3c83fe730036df781436136721e2dcd8f08c540e4470d255eeedd5f300f2d221" Sep 10 16:31:34.030103 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:34.030063 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4"] Sep 10 16:31:34.035385 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:34.035347 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-90bd7e-openshift-default-68f8797466-5bzg4"] Sep 10 16:31:34.981391 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:34.981348 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" path="/var/lib/kubelet/pods/aeef5b3d-8ba8-4901-a1d8-9135ec7b5092/volumes" Sep 10 16:31:41.131735 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:41.131707 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/1.log" Sep 10 16:31:41.136073 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:41.136042 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/1.log" Sep 10 16:31:41.137740 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:41.137712 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4c6j_2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd/ovn-acl-logging/0.log" Sep 10 16:31:41.142422 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:41.142394 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4c6j_2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd/ovn-acl-logging/0.log" Sep 10 16:31:43.979367 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:43.979325 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:46.040281 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:46.040237 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:50.573128 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:50.573082 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:31:54.827394 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:31:54.827355 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:06.553045 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:06.553001 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:32:12.151484 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:12.151449 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-85d2w_c9c7e7f6-7f76-493c-aa7a-f69a70592836/payload-processing/0.log" Sep 10 16:32:12.625491 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:12.625451 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:12.929575 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:12.929465 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq"] Sep 10 16:32:12.930320 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:12.930279 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d4a1a29-be5b-4320-8de2-818b6535533d" containerName="revoke-keys" Sep 10 16:32:12.930320 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:12.930322 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d4a1a29-be5b-4320-8de2-818b6535533d" containerName="revoke-keys" Sep 10 16:32:12.930518 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:12.930342 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" containerName="istio-proxy" Sep 10 16:32:12.930518 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:12.930352 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" containerName="istio-proxy" Sep 10 16:32:12.930518 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:12.930377 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fbc7294-dcdd-4246-8142-0db96e7f6ab8" containerName="payload-pre-processing" Sep 10 16:32:12.930518 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:12.930388 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fbc7294-dcdd-4246-8142-0db96e7f6ab8" containerName="payload-pre-processing" Sep 10 16:32:12.930518 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:12.930496 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" containerName="istio-proxy" Sep 10 16:32:12.930518 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:12.930506 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fbc7294-dcdd-4246-8142-0db96e7f6ab8" containerName="payload-pre-processing" Sep 10 16:32:12.930518 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:12.930521 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d4a1a29-be5b-4320-8de2-818b6535533d" containerName="revoke-keys" Sep 10 16:32:12.933978 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:12.933952 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:12.936968 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:12.936937 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-68ca72-openshift-default-dockercfg-bxlr6\"" Sep 10 16:32:12.946057 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:12.946018 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq"] Sep 10 16:32:12.965663 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:12.965630 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:13.014451 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.014404 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-podinfo\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.014677 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.014491 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istiod-ca-cert\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.014677 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.014520 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-workload-socket\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.014677 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.014629 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-workload-certs\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.014677 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.014655 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-token\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.014929 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.014719 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-envoy\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.014929 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.014796 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-data\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.014929 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.014818 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-credential-socket\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.014929 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.014886 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtrth\" (UniqueName: \"kubernetes.io/projected/d8982141-a6d5-4c7f-ac15-99b81ff78d40-kube-api-access-dtrth\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.116261 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.116211 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-envoy\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.116470 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.116277 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-data\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.116470 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.116432 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-credential-socket\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.116563 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.116504 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtrth\" (UniqueName: \"kubernetes.io/projected/d8982141-a6d5-4c7f-ac15-99b81ff78d40-kube-api-access-dtrth\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.116563 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.116534 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-podinfo\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.116670 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.116585 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istiod-ca-cert\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.116670 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.116617 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-workload-socket\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.116761 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.116676 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-data\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.116761 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.116699 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-workload-certs\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.116761 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.116729 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-token\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.116901 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.116832 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-credential-socket\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.117002 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.116979 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-workload-socket\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.117105 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.117078 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-workload-certs\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.117334 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.117289 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istiod-ca-cert\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.118938 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.118908 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-envoy\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.119286 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.119263 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-podinfo\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.126928 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.126881 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-token\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.127200 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.127174 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtrth\" (UniqueName: \"kubernetes.io/projected/d8982141-a6d5-4c7f-ac15-99b81ff78d40-kube-api-access-dtrth\") pod \"e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.223051 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.222952 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:13.247590 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.247551 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:13.536012 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.535978 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq"] Sep 10 16:32:13.537895 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:32:13.537858 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8982141_a6d5_4c7f_ac15_99b81ff78d40.slice/crio-0275d3e759ab7428cb8492f36435e56f0b9a57beded3cd886fc150270b8730e0 WatchSource:0}: Error finding container 0275d3e759ab7428cb8492f36435e56f0b9a57beded3cd886fc150270b8730e0: Status 404 returned error can't find the container with id 0275d3e759ab7428cb8492f36435e56f0b9a57beded3cd886fc150270b8730e0 Sep 10 16:32:13.540898 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.540857 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:32:13.540979 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.540941 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:32:13.540979 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:13.540971 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:32:14.268554 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:14.268517 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" event={"ID":"d8982141-a6d5-4c7f-ac15-99b81ff78d40","Type":"ContainerStarted","Data":"dc4b75c9abefd0e9fb0ea4bda989f6b16a08acb1da6a09d94d16860526912728"} Sep 10 16:32:14.269024 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:14.268585 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" event={"ID":"d8982141-a6d5-4c7f-ac15-99b81ff78d40","Type":"ContainerStarted","Data":"0275d3e759ab7428cb8492f36435e56f0b9a57beded3cd886fc150270b8730e0"} Sep 10 16:32:14.295503 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:14.295444 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" podStartSLOduration=2.295421034 podStartE2EDuration="2.295421034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:32:14.291876363 +0000 UTC m=+1833.815121559" watchObservedRunningTime="2026-09-10 16:32:14.295421034 +0000 UTC m=+1833.818666227" Sep 10 16:32:15.248766 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:15.248721 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:15.254805 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:15.254771 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:15.274990 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:15.274956 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:15.275993 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:15.275966 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:16.290226 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:16.290160 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:16.382805 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:16.382766 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:17.621622 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:17.621581 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:17.719522 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:17.719483 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:18.143801 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:18.143754 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:18.239681 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:18.239646 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:18.335119 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:18.335087 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:27.856679 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:27.856627 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:32:39.456310 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:39.456258 2541 generic.go:356] "Generic (PLEG): container finished" podID="d8982141-a6d5-4c7f-ac15-99b81ff78d40" containerID="dc4b75c9abefd0e9fb0ea4bda989f6b16a08acb1da6a09d94d16860526912728" exitCode=0 Sep 10 16:32:39.456757 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:39.456335 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" event={"ID":"d8982141-a6d5-4c7f-ac15-99b81ff78d40","Type":"ContainerDied","Data":"dc4b75c9abefd0e9fb0ea4bda989f6b16a08acb1da6a09d94d16860526912728"} Sep 10 16:32:39.456803 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:39.456770 2541 scope.go:117] "RemoveContainer" containerID="dc4b75c9abefd0e9fb0ea4bda989f6b16a08acb1da6a09d94d16860526912728" Sep 10 16:32:39.458241 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:39.458206 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:32:39.458347 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:39.458285 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:32:39.458347 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:39.458333 2541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 16:32:40.465018 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:40.464980 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" event={"ID":"d8982141-a6d5-4c7f-ac15-99b81ff78d40","Type":"ContainerStarted","Data":"d5eb54f22109849a67d146ef651e44545e5caf2e4fa130d5bfd9ed8f9baca0fc"} Sep 10 16:32:41.248503 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:41.248459 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:41.249739 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:41.249712 2541 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Sep 10 16:32:41.249832 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:41.249771 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" podUID="d8982141-a6d5-4c7f-ac15-99b81ff78d40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Sep 10 16:32:41.319802 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:41.319755 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:42.031222 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:42.031171 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-68ca72-hdgnk"] Sep 10 16:32:42.031831 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:42.031807 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" containerName="istio-proxy" Sep 10 16:32:42.031913 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:42.031834 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" containerName="istio-proxy" Sep 10 16:32:42.031973 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:42.031949 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeef5b3d-8ba8-4901-a1d8-9135ec7b5092" containerName="istio-proxy" Sep 10 16:32:42.035410 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:42.035380 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-68ca72-hdgnk" Sep 10 16:32:42.037609 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:42.037582 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:32:42.037771 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:42.037631 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l2dj2\"" Sep 10 16:32:42.042533 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:42.042498 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-68ca72-hdgnk"] Sep 10 16:32:42.116112 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:42.116066 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a641803-8cb3-4124-b9c4-27e0ef39dbd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-68ca72-hdgnk\" (UID: \"0a641803-8cb3-4124-b9c4-27e0ef39dbd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-68ca72-hdgnk" Sep 10 16:32:42.217613 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:42.217566 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a641803-8cb3-4124-b9c4-27e0ef39dbd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-68ca72-hdgnk\" (UID: \"0a641803-8cb3-4124-b9c4-27e0ef39dbd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-68ca72-hdgnk" Sep 10 16:32:42.218284 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:42.218260 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a641803-8cb3-4124-b9c4-27e0ef39dbd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-68ca72-hdgnk\" (UID: \"0a641803-8cb3-4124-b9c4-27e0ef39dbd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-68ca72-hdgnk" Sep 10 16:32:42.248843 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:42.248804 2541 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Sep 10 16:32:42.248999 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:42.248884 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" podUID="d8982141-a6d5-4c7f-ac15-99b81ff78d40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Sep 10 16:32:42.348697 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:42.348593 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-68ca72-hdgnk" Sep 10 16:32:42.656848 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:42.656817 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-68ca72-hdgnk"] Sep 10 16:32:42.658944 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:32:42.658909 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a641803_8cb3_4124_b9c4_27e0ef39dbd5.slice/crio-7f8e42e8fd0a4ac7d64c638d3b5d27a720ff13ad7f967a6640e4b73116a569e3 WatchSource:0}: Error finding container 7f8e42e8fd0a4ac7d64c638d3b5d27a720ff13ad7f967a6640e4b73116a569e3: Status 404 returned error can't find the container with id 7f8e42e8fd0a4ac7d64c638d3b5d27a720ff13ad7f967a6640e4b73116a569e3 Sep 10 16:32:43.248807 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:43.248625 2541 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Sep 10 16:32:43.248807 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:43.248689 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" podUID="d8982141-a6d5-4c7f-ac15-99b81ff78d40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Sep 10 16:32:43.490582 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:43.490530 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-68ca72-hdgnk" event={"ID":"0a641803-8cb3-4124-b9c4-27e0ef39dbd5","Type":"ContainerStarted","Data":"5ed8ef20680e2d11f2412739bd716375e0e75b633a2325e359b7150cce361806"} Sep 10 16:32:43.490813 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:43.490609 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-68ca72-hdgnk" event={"ID":"0a641803-8cb3-4124-b9c4-27e0ef39dbd5","Type":"ContainerStarted","Data":"7f8e42e8fd0a4ac7d64c638d3b5d27a720ff13ad7f967a6640e4b73116a569e3"} Sep 10 16:32:43.510224 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:43.510151 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-68ca72-hdgnk" podStartSLOduration=1.510124112 podStartE2EDuration="1.510124112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:32:43.506695876 +0000 UTC m=+1863.029941110" watchObservedRunningTime="2026-09-10 16:32:43.510124112 +0000 UTC m=+1863.033369310" Sep 10 16:32:44.248143 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:44.248106 2541 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Sep 10 16:32:44.248360 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:44.248172 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" podUID="d8982141-a6d5-4c7f-ac15-99b81ff78d40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Sep 10 16:32:44.507088 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:44.506984 2541 generic.go:356] "Generic (PLEG): container finished" podID="0a641803-8cb3-4124-b9c4-27e0ef39dbd5" containerID="5ed8ef20680e2d11f2412739bd716375e0e75b633a2325e359b7150cce361806" exitCode=0 Sep 10 16:32:44.507088 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:44.507064 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-68ca72-hdgnk" event={"ID":"0a641803-8cb3-4124-b9c4-27e0ef39dbd5","Type":"ContainerDied","Data":"5ed8ef20680e2d11f2412739bd716375e0e75b633a2325e359b7150cce361806"} Sep 10 16:32:45.247935 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:45.247902 2541 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Sep 10 16:32:45.248131 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:45.247956 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" podUID="d8982141-a6d5-4c7f-ac15-99b81ff78d40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Sep 10 16:32:45.275318 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:45.275267 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:45.672056 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:45.672028 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-68ca72-hdgnk" Sep 10 16:32:45.755703 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:45.755662 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a641803-8cb3-4124-b9c4-27e0ef39dbd5-maas-api-service-ca\") pod \"0a641803-8cb3-4124-b9c4-27e0ef39dbd5\" (UID: \"0a641803-8cb3-4124-b9c4-27e0ef39dbd5\") " Sep 10 16:32:45.756031 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:45.756004 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a641803-8cb3-4124-b9c4-27e0ef39dbd5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0a641803-8cb3-4124-b9c4-27e0ef39dbd5" (UID: "0a641803-8cb3-4124-b9c4-27e0ef39dbd5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:32:45.856655 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:45.856548 2541 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a641803-8cb3-4124-b9c4-27e0ef39dbd5-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:32:46.248049 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:46.248001 2541 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Sep 10 16:32:46.248242 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:46.248062 2541 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" podUID="d8982141-a6d5-4c7f-ac15-99b81ff78d40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Sep 10 16:32:46.522072 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:46.521971 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-68ca72-hdgnk" event={"ID":"0a641803-8cb3-4124-b9c4-27e0ef39dbd5","Type":"ContainerDied","Data":"7f8e42e8fd0a4ac7d64c638d3b5d27a720ff13ad7f967a6640e4b73116a569e3"} Sep 10 16:32:46.522072 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:46.522022 2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f8e42e8fd0a4ac7d64c638d3b5d27a720ff13ad7f967a6640e4b73116a569e3" Sep 10 16:32:46.522072 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:46.521984 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-68ca72-hdgnk" Sep 10 16:32:47.252568 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:47.252534 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:47.253725 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:47.253700 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:32:48.085538 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:48.085499 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:48.189239 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:48.189207 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:48.272167 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:48.272132 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:48.330629 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:48.330579 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-68ca72-hdgnk"] Sep 10 16:32:48.335852 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:48.335747 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-68ca72-hdgnk"] Sep 10 16:32:48.372097 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:48.372064 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:48.731877 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:48.731836 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:48.980792 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:48.980745 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a641803-8cb3-4124-b9c4-27e0ef39dbd5" path="/var/lib/kubelet/pods/0a641803-8cb3-4124-b9c4-27e0ef39dbd5/volumes" Sep 10 16:32:52.953983 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:52.953936 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:32:53.657911 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:53.657873 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:55.154513 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:55.154479 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:55.232096 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:55.232060 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:55.454682 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:55.454586 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq"] Sep 10 16:32:55.454839 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:55.454803 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" podUID="d8982141-a6d5-4c7f-ac15-99b81ff78d40" containerName="istio-proxy" containerID="cri-o://d5eb54f22109849a67d146ef651e44545e5caf2e4fa130d5bfd9ed8f9baca0fc" gracePeriod=30 Sep 10 16:32:55.512791 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:55.512732 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:59.232216 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:59.232158 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-814d40-cj4v2"] Sep 10 16:32:59.233157 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:59.233125 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a641803-8cb3-4124-b9c4-27e0ef39dbd5" containerName="revoke-keys" Sep 10 16:32:59.233315 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:59.233283 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a641803-8cb3-4124-b9c4-27e0ef39dbd5" containerName="revoke-keys" Sep 10 16:32:59.233673 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:59.233657 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a641803-8cb3-4124-b9c4-27e0ef39dbd5" containerName="revoke-keys" Sep 10 16:32:59.239403 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:59.239367 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-814d40-cj4v2" Sep 10 16:32:59.240487 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:59.240460 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-814d40-cj4v2"] Sep 10 16:32:59.241896 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:59.241860 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:32:59.242056 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:59.241860 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l2dj2\"" Sep 10 16:32:59.297899 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:59.297853 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d875b97-c698-4be3-9020-1f6332e4e9f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-814d40-cj4v2\" (UID: \"0d875b97-c698-4be3-9020-1f6332e4e9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-814d40-cj4v2" Sep 10 16:32:59.398647 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:59.398601 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d875b97-c698-4be3-9020-1f6332e4e9f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-814d40-cj4v2\" (UID: \"0d875b97-c698-4be3-9020-1f6332e4e9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-814d40-cj4v2" Sep 10 16:32:59.399440 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:59.399414 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d875b97-c698-4be3-9020-1f6332e4e9f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-814d40-cj4v2\" (UID: \"0d875b97-c698-4be3-9020-1f6332e4e9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-814d40-cj4v2" Sep 10 16:32:59.435995 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:59.435957 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:59.552088 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:59.551983 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-814d40-cj4v2" Sep 10 16:32:59.835847 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:32:59.835815 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-814d40-cj4v2"] Sep 10 16:32:59.837870 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:32:59.837833 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d875b97_c698_4be3_9020_1f6332e4e9f8.slice/crio-2638714acea3b5bfbeb9bd11d31cba6fe49c7c236cfdb9905a670a770c46c210 WatchSource:0}: Error finding container 2638714acea3b5bfbeb9bd11d31cba6fe49c7c236cfdb9905a670a770c46c210: Status 404 returned error can't find the container with id 2638714acea3b5bfbeb9bd11d31cba6fe49c7c236cfdb9905a670a770c46c210 Sep 10 16:33:00.617789 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.617748 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-814d40-cj4v2" event={"ID":"0d875b97-c698-4be3-9020-1f6332e4e9f8","Type":"ContainerStarted","Data":"fb4d5f235da97f054fa0e72dd01e0d78171207cd10860120f26c8b4e337ac2f7"} Sep 10 16:33:00.618347 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.617824 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-814d40-cj4v2" event={"ID":"0d875b97-c698-4be3-9020-1f6332e4e9f8","Type":"ContainerStarted","Data":"2638714acea3b5bfbeb9bd11d31cba6fe49c7c236cfdb9905a670a770c46c210"} Sep 10 16:33:00.620291 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.620259 2541 generic.go:356] "Generic (PLEG): container finished" podID="d8982141-a6d5-4c7f-ac15-99b81ff78d40" containerID="d5eb54f22109849a67d146ef651e44545e5caf2e4fa130d5bfd9ed8f9baca0fc" exitCode=0 Sep 10 16:33:00.620458 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.620335 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" event={"ID":"d8982141-a6d5-4c7f-ac15-99b81ff78d40","Type":"ContainerDied","Data":"d5eb54f22109849a67d146ef651e44545e5caf2e4fa130d5bfd9ed8f9baca0fc"} Sep 10 16:33:00.620458 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.620409 2541 scope.go:117] "RemoveContainer" containerID="dc4b75c9abefd0e9fb0ea4bda989f6b16a08acb1da6a09d94d16860526912728" Sep 10 16:33:00.645934 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.645846 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-814d40-cj4v2" podStartSLOduration=1.645822553 podStartE2EDuration="1.645822553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:33:00.643412025 +0000 UTC m=+1880.166657223" watchObservedRunningTime="2026-09-10 16:33:00.645822553 +0000 UTC m=+1880.169067759" Sep 10 16:33:00.751452 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.751423 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:33:00.812398 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.812366 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-workload-certs\") pod \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " Sep 10 16:33:00.812592 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.812408 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtrth\" (UniqueName: \"kubernetes.io/projected/d8982141-a6d5-4c7f-ac15-99b81ff78d40-kube-api-access-dtrth\") pod \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " Sep 10 16:33:00.812592 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.812442 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-envoy\") pod \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " Sep 10 16:33:00.812592 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.812467 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-credential-socket\") pod \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " Sep 10 16:33:00.812592 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.812512 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istiod-ca-cert\") pod \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " Sep 10 16:33:00.812592 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.812560 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-workload-socket\") pod \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " Sep 10 16:33:00.812860 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.812629 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-data\") pod \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " Sep 10 16:33:00.812860 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.812664 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-podinfo\") pod \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " Sep 10 16:33:00.812860 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.812677 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d8982141-a6d5-4c7f-ac15-99b81ff78d40" (UID: "d8982141-a6d5-4c7f-ac15-99b81ff78d40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:33:00.812860 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.812702 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-token\") pod \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\" (UID: \"d8982141-a6d5-4c7f-ac15-99b81ff78d40\") " Sep 10 16:33:00.813066 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.812864 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d8982141-a6d5-4c7f-ac15-99b81ff78d40" (UID: "d8982141-a6d5-4c7f-ac15-99b81ff78d40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:33:00.813066 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.813041 2541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-credential-socket\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:33:00.813066 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.813059 2541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-workload-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:33:00.813219 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.813120 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d8982141-a6d5-4c7f-ac15-99b81ff78d40" (UID: "d8982141-a6d5-4c7f-ac15-99b81ff78d40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:33:00.813716 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.813540 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d8982141-a6d5-4c7f-ac15-99b81ff78d40" (UID: "d8982141-a6d5-4c7f-ac15-99b81ff78d40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:33:00.813716 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.813614 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-data" (OuterVolumeSpecName: "istio-data") pod "d8982141-a6d5-4c7f-ac15-99b81ff78d40" (UID: "d8982141-a6d5-4c7f-ac15-99b81ff78d40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:33:00.815156 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.815111 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d8982141-a6d5-4c7f-ac15-99b81ff78d40" (UID: "d8982141-a6d5-4c7f-ac15-99b81ff78d40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:33:00.815284 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.815194 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d8982141-a6d5-4c7f-ac15-99b81ff78d40" (UID: "d8982141-a6d5-4c7f-ac15-99b81ff78d40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:33:00.815531 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.815507 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8982141-a6d5-4c7f-ac15-99b81ff78d40-kube-api-access-dtrth" (OuterVolumeSpecName: "kube-api-access-dtrth") pod "d8982141-a6d5-4c7f-ac15-99b81ff78d40" (UID: "d8982141-a6d5-4c7f-ac15-99b81ff78d40"). InnerVolumeSpecName "kube-api-access-dtrth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:33:00.815626 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.815601 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-token" (OuterVolumeSpecName: "istio-token") pod "d8982141-a6d5-4c7f-ac15-99b81ff78d40" (UID: "d8982141-a6d5-4c7f-ac15-99b81ff78d40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:33:00.914180 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.914134 2541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-podinfo\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:33:00.914180 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.914171 2541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-token\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:33:00.914180 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.914181 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtrth\" (UniqueName: \"kubernetes.io/projected/d8982141-a6d5-4c7f-ac15-99b81ff78d40-kube-api-access-dtrth\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:33:00.914180 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.914192 2541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-envoy\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:33:00.914508 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.914200 2541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istiod-ca-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:33:00.914508 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.914207 2541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-workload-socket\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:33:00.914508 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:00.914216 2541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8982141-a6d5-4c7f-ac15-99b81ff78d40-istio-data\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:33:01.628387 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:01.628360 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" Sep 10 16:33:01.628846 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:01.628359 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq" event={"ID":"d8982141-a6d5-4c7f-ac15-99b81ff78d40","Type":"ContainerDied","Data":"0275d3e759ab7428cb8492f36435e56f0b9a57beded3cd886fc150270b8730e0"} Sep 10 16:33:01.628846 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:01.628493 2541 scope.go:117] "RemoveContainer" containerID="d5eb54f22109849a67d146ef651e44545e5caf2e4fa130d5bfd9ed8f9baca0fc" Sep 10 16:33:01.630407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:01.630357 2541 generic.go:356] "Generic (PLEG): container finished" podID="0d875b97-c698-4be3-9020-1f6332e4e9f8" containerID="fb4d5f235da97f054fa0e72dd01e0d78171207cd10860120f26c8b4e337ac2f7" exitCode=0 Sep 10 16:33:01.630535 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:01.630484 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-814d40-cj4v2" event={"ID":"0d875b97-c698-4be3-9020-1f6332e4e9f8","Type":"ContainerDied","Data":"fb4d5f235da97f054fa0e72dd01e0d78171207cd10860120f26c8b4e337ac2f7"} Sep 10 16:33:01.650172 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:01.650133 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq"] Sep 10 16:33:01.655080 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:01.655045 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-68ca72-openshift-default-5f7f6bfc4b-sf5wq"] Sep 10 16:33:02.800387 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:02.800355 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-814d40-cj4v2" Sep 10 16:33:02.933951 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:02.933838 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d875b97-c698-4be3-9020-1f6332e4e9f8-maas-api-service-ca\") pod \"0d875b97-c698-4be3-9020-1f6332e4e9f8\" (UID: \"0d875b97-c698-4be3-9020-1f6332e4e9f8\") " Sep 10 16:33:02.934234 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:02.934211 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d875b97-c698-4be3-9020-1f6332e4e9f8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0d875b97-c698-4be3-9020-1f6332e4e9f8" (UID: "0d875b97-c698-4be3-9020-1f6332e4e9f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:33:02.979707 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:02.979670 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8982141-a6d5-4c7f-ac15-99b81ff78d40" path="/var/lib/kubelet/pods/d8982141-a6d5-4c7f-ac15-99b81ff78d40/volumes" Sep 10 16:33:03.035536 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:03.035492 2541 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d875b97-c698-4be3-9020-1f6332e4e9f8-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:33:03.298256 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:03.298168 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:03.645751 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:03.645721 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-814d40-cj4v2" Sep 10 16:33:03.645751 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:03.645725 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-814d40-cj4v2" event={"ID":"0d875b97-c698-4be3-9020-1f6332e4e9f8","Type":"ContainerDied","Data":"2638714acea3b5bfbeb9bd11d31cba6fe49c7c236cfdb9905a670a770c46c210"} Sep 10 16:33:03.646028 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:03.645772 2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2638714acea3b5bfbeb9bd11d31cba6fe49c7c236cfdb9905a670a770c46c210" Sep 10 16:33:12.848337 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:12.848276 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:33:14.963678 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:14.963632 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:15.063468 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:15.063426 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:15.166035 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:15.165997 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:15.167940 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:15.167908 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-814d40-cj4v2"] Sep 10 16:33:15.171488 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:15.171449 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-814d40-cj4v2"] Sep 10 16:33:15.264776 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:15.264683 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:15.575012 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:15.574916 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:16.982533 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:16.982486 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d875b97-c698-4be3-9020-1f6332e4e9f8" path="/var/lib/kubelet/pods/0d875b97-c698-4be3-9020-1f6332e4e9f8/volumes" Sep 10 16:33:20.653508 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:20.653474 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:25.358202 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:25.358157 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:33:36.564957 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:36.564918 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:38.259095 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:38.259059 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:56.901645 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:56.901604 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:58.101985 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:33:58.101953 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:04.406468 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:04.406433 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:04.492394 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:04.492353 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:04.700628 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:04.700525 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:05.078946 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:05.078843 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-275a06-fsmxr"] Sep 10 16:34:05.079280 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:05.079265 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8982141-a6d5-4c7f-ac15-99b81ff78d40" containerName="istio-proxy" Sep 10 16:34:05.079280 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:05.079281 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8982141-a6d5-4c7f-ac15-99b81ff78d40" containerName="istio-proxy" Sep 10 16:34:05.079434 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:05.079294 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d875b97-c698-4be3-9020-1f6332e4e9f8" containerName="revoke-keys" Sep 10 16:34:05.079434 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:05.079314 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d875b97-c698-4be3-9020-1f6332e4e9f8" containerName="revoke-keys" Sep 10 16:34:05.079434 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:05.079323 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8982141-a6d5-4c7f-ac15-99b81ff78d40" containerName="istio-proxy" Sep 10 16:34:05.079434 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:05.079328 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8982141-a6d5-4c7f-ac15-99b81ff78d40" containerName="istio-proxy" Sep 10 16:34:05.079434 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:05.079396 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8982141-a6d5-4c7f-ac15-99b81ff78d40" containerName="istio-proxy" Sep 10 16:34:05.079434 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:05.079405 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d875b97-c698-4be3-9020-1f6332e4e9f8" containerName="revoke-keys" Sep 10 16:34:05.079434 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:05.079412 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8982141-a6d5-4c7f-ac15-99b81ff78d40" containerName="istio-proxy" Sep 10 16:34:05.082786 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:05.082760 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-275a06-fsmxr" Sep 10 16:34:05.085215 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:05.085189 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:34:05.085427 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:05.085251 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l2dj2\"" Sep 10 16:34:05.092221 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:05.092176 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-275a06-fsmxr"] Sep 10 16:34:05.148511 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:05.148469 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-275a06-fsmxr\" (UID: \"0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-275a06-fsmxr" Sep 10 16:34:05.249202 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:05.249162 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-275a06-fsmxr\" (UID: \"0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-275a06-fsmxr" Sep 10 16:34:05.249830 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:05.249808 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-275a06-fsmxr\" (UID: \"0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-275a06-fsmxr" Sep 10 16:34:05.323544 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:05.323501 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:05.398812 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:05.398768 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-275a06-fsmxr" Sep 10 16:34:05.679487 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:05.679454 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-275a06-fsmxr"] Sep 10 16:34:05.682579 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:34:05.682543 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf7a7a1_5ac6_4e7c_9474_11d6444fc0a2.slice/crio-03a1080a679cd63a991c1a6365aa3a2626703447fbcf72f2cb4d37ede929395e WatchSource:0}: Error finding container 03a1080a679cd63a991c1a6365aa3a2626703447fbcf72f2cb4d37ede929395e: Status 404 returned error can't find the container with id 03a1080a679cd63a991c1a6365aa3a2626703447fbcf72f2cb4d37ede929395e Sep 10 16:34:06.050588 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:06.050555 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-275a06-fsmxr" event={"ID":"0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2","Type":"ContainerStarted","Data":"d10b72980955c18d6ae439f2e9617c31c9992c3c829a6ac307da87c403168206"} Sep 10 16:34:06.050785 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:06.050609 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-275a06-fsmxr" event={"ID":"0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2","Type":"ContainerStarted","Data":"03a1080a679cd63a991c1a6365aa3a2626703447fbcf72f2cb4d37ede929395e"} Sep 10 16:34:06.065581 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:06.065522 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-275a06-fsmxr" podStartSLOduration=1.065499793 podStartE2EDuration="1.065499793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:34:06.064349492 +0000 UTC m=+1945.587594689" watchObservedRunningTime="2026-09-10 16:34:06.065499793 +0000 UTC m=+1945.588744989" Sep 10 16:34:07.058286 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:07.058248 2541 generic.go:356] "Generic (PLEG): container finished" podID="0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2" containerID="d10b72980955c18d6ae439f2e9617c31c9992c3c829a6ac307da87c403168206" exitCode=0 Sep 10 16:34:07.058736 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:07.058337 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-275a06-fsmxr" event={"ID":"0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2","Type":"ContainerDied","Data":"d10b72980955c18d6ae439f2e9617c31c9992c3c829a6ac307da87c403168206"} Sep 10 16:34:08.207238 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:08.207205 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-275a06-fsmxr" Sep 10 16:34:08.282047 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:08.281999 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2-maas-api-service-ca\") pod \"0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2\" (UID: \"0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2\") " Sep 10 16:34:08.282421 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:08.282396 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2" (UID: "0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:34:08.383160 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:08.383121 2541 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:34:08.385582 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:08.385543 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:09.072059 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:09.072024 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-275a06-fsmxr" Sep 10 16:34:09.072059 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:09.072039 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-275a06-fsmxr" event={"ID":"0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2","Type":"ContainerDied","Data":"03a1080a679cd63a991c1a6365aa3a2626703447fbcf72f2cb4d37ede929395e"} Sep 10 16:34:09.072276 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:09.072091 2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03a1080a679cd63a991c1a6365aa3a2626703447fbcf72f2cb4d37ede929395e" Sep 10 16:34:15.332094 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:15.332053 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:15.963794 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:15.963763 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:16.010435 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:16.010396 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc"] Sep 10 16:34:16.010692 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:16.010663 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc" podUID="bad02186-02e9-4565-9c18-34a59d63a36e" containerName="payload-pre-processing" containerID="cri-o://46de7140380a5b0c4e7d0869b246c83c011c0d573db091039c6c3fd5b259770c" gracePeriod=30 Sep 10 16:34:16.075809 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:16.075774 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:16.183731 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:16.183698 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:16.278628 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:16.278600 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc" Sep 10 16:34:16.283808 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:16.283770 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-275a06-fsmxr"] Sep 10 16:34:16.287744 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:16.287706 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-275a06-fsmxr"] Sep 10 16:34:16.289759 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:16.289727 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:16.370723 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:16.370689 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad02186-02e9-4565-9c18-34a59d63a36e-plugins-config-volume\") pod \"bad02186-02e9-4565-9c18-34a59d63a36e\" (UID: \"bad02186-02e9-4565-9c18-34a59d63a36e\") " Sep 10 16:34:16.371209 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:16.370734 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hgs5\" (UniqueName: \"kubernetes.io/projected/bad02186-02e9-4565-9c18-34a59d63a36e-kube-api-access-4hgs5\") pod \"bad02186-02e9-4565-9c18-34a59d63a36e\" (UID: \"bad02186-02e9-4565-9c18-34a59d63a36e\") " Sep 10 16:34:16.371209 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:16.371159 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad02186-02e9-4565-9c18-34a59d63a36e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bad02186-02e9-4565-9c18-34a59d63a36e" (UID: "bad02186-02e9-4565-9c18-34a59d63a36e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:34:16.373615 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:16.373581 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad02186-02e9-4565-9c18-34a59d63a36e-kube-api-access-4hgs5" (OuterVolumeSpecName: "kube-api-access-4hgs5") pod "bad02186-02e9-4565-9c18-34a59d63a36e" (UID: "bad02186-02e9-4565-9c18-34a59d63a36e"). InnerVolumeSpecName "kube-api-access-4hgs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:34:16.376705 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:16.376675 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:16.472339 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:16.472248 2541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad02186-02e9-4565-9c18-34a59d63a36e-plugins-config-volume\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:34:16.472339 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:16.472334 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hgs5\" (UniqueName: \"kubernetes.io/projected/bad02186-02e9-4565-9c18-34a59d63a36e-kube-api-access-4hgs5\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:34:16.670754 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:16.670712 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:16.980236 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:16.980152 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2" path="/var/lib/kubelet/pods/0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2/volumes" Sep 10 16:34:17.128054 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:17.128017 2541 generic.go:356] "Generic (PLEG): container finished" podID="bad02186-02e9-4565-9c18-34a59d63a36e" containerID="46de7140380a5b0c4e7d0869b246c83c011c0d573db091039c6c3fd5b259770c" exitCode=0 Sep 10 16:34:17.128054 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:17.128065 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc" Sep 10 16:34:17.128327 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:17.128081 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc" event={"ID":"bad02186-02e9-4565-9c18-34a59d63a36e","Type":"ContainerDied","Data":"46de7140380a5b0c4e7d0869b246c83c011c0d573db091039c6c3fd5b259770c"} Sep 10 16:34:17.128327 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:17.128128 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc" event={"ID":"bad02186-02e9-4565-9c18-34a59d63a36e","Type":"ContainerDied","Data":"43f8d280e9d5ba6ba4d268ff245d5999cd8696a85adc3b8a4c2f0e69e3865335"} Sep 10 16:34:17.128327 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:17.128153 2541 scope.go:117] "RemoveContainer" containerID="46de7140380a5b0c4e7d0869b246c83c011c0d573db091039c6c3fd5b259770c" Sep 10 16:34:17.138851 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:17.138826 2541 scope.go:117] "RemoveContainer" containerID="46de7140380a5b0c4e7d0869b246c83c011c0d573db091039c6c3fd5b259770c" Sep 10 16:34:17.139281 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:34:17.139250 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46de7140380a5b0c4e7d0869b246c83c011c0d573db091039c6c3fd5b259770c\": container with ID starting with 46de7140380a5b0c4e7d0869b246c83c011c0d573db091039c6c3fd5b259770c not found: ID does not exist" containerID="46de7140380a5b0c4e7d0869b246c83c011c0d573db091039c6c3fd5b259770c" Sep 10 16:34:17.139464 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:17.139291 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46de7140380a5b0c4e7d0869b246c83c011c0d573db091039c6c3fd5b259770c"} err="failed to get container status \"46de7140380a5b0c4e7d0869b246c83c011c0d573db091039c6c3fd5b259770c\": rpc error: code = NotFound desc = could not find container \"46de7140380a5b0c4e7d0869b246c83c011c0d573db091039c6c3fd5b259770c\": container with ID starting with 46de7140380a5b0c4e7d0869b246c83c011c0d573db091039c6c3fd5b259770c not found: ID does not exist" Sep 10 16:34:17.167613 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:17.167574 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc"] Sep 10 16:34:17.170231 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:17.170201 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-275a06-79d8f75dc4-2pgnc"] Sep 10 16:34:18.980022 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:18.979967 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bad02186-02e9-4565-9c18-34a59d63a36e" path="/var/lib/kubelet/pods/bad02186-02e9-4565-9c18-34a59d63a36e/volumes" Sep 10 16:34:19.467143 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:19.467098 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:34:22.683572 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:22.683533 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:22.790900 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:22.790857 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:23.017404 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:23.017289 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk"] Sep 10 16:34:23.017589 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:23.017526 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" podUID="00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" containerName="istio-proxy" containerID="cri-o://6a3b6951f686f4ee14d636396c0909146e01f613fa2c79fa87f10620747a31c3" gracePeriod=30 Sep 10 16:34:23.020181 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:23.020149 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:23.463400 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:23.463356 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:28.211746 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.211683 2541 generic.go:356] "Generic (PLEG): container finished" podID="00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" containerID="6a3b6951f686f4ee14d636396c0909146e01f613fa2c79fa87f10620747a31c3" exitCode=0 Sep 10 16:34:28.214463 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.211899 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" event={"ID":"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060","Type":"ContainerDied","Data":"6a3b6951f686f4ee14d636396c0909146e01f613fa2c79fa87f10620747a31c3"} Sep 10 16:34:28.214463 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.211994 2541 scope.go:117] "RemoveContainer" containerID="2266be7fb8a9ff53af77dbf806db6e4c29f74d48c4b60b38cc89d486c57f9e55" Sep 10 16:34:28.285431 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.285347 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:34:28.396282 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.396231 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-workload-socket\") pod \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " Sep 10 16:34:28.396505 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.396324 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xqqt\" (UniqueName: \"kubernetes.io/projected/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-kube-api-access-5xqqt\") pod \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " Sep 10 16:34:28.396505 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.396359 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-credential-socket\") pod \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " Sep 10 16:34:28.396505 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.396385 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istiod-ca-cert\") pod \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " Sep 10 16:34:28.396505 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.396419 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-envoy\") pod \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " Sep 10 16:34:28.396505 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.396452 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-token\") pod \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " Sep 10 16:34:28.396786 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.396544 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-workload-certs\") pod \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " Sep 10 16:34:28.396786 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.396568 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-data\") pod \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " Sep 10 16:34:28.396786 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.396607 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-podinfo\") pod \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\" (UID: \"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060\") " Sep 10 16:34:28.396786 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.396670 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" (UID: "00f0a5c3-cd3a-4186-a3fd-df5ca7b08060"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:28.396786 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.396686 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" (UID: "00f0a5c3-cd3a-4186-a3fd-df5ca7b08060"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:28.397045 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.396896 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" (UID: "00f0a5c3-cd3a-4186-a3fd-df5ca7b08060"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:28.397045 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.396981 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-data" (OuterVolumeSpecName: "istio-data") pod "00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" (UID: "00f0a5c3-cd3a-4186-a3fd-df5ca7b08060"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:28.397045 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.397030 2541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-workload-socket\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:34:28.397191 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.397048 2541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-credential-socket\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:34:28.397191 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.397057 2541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-workload-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:34:28.397417 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.397345 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" (UID: "00f0a5c3-cd3a-4186-a3fd-df5ca7b08060"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:34:28.399034 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.399004 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-kube-api-access-5xqqt" (OuterVolumeSpecName: "kube-api-access-5xqqt") pod "00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" (UID: "00f0a5c3-cd3a-4186-a3fd-df5ca7b08060"). InnerVolumeSpecName "kube-api-access-5xqqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:34:28.399164 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.399049 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" (UID: "00f0a5c3-cd3a-4186-a3fd-df5ca7b08060"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:28.399213 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.399162 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" (UID: "00f0a5c3-cd3a-4186-a3fd-df5ca7b08060"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:34:28.399213 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.399167 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-token" (OuterVolumeSpecName: "istio-token") pod "00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" (UID: "00f0a5c3-cd3a-4186-a3fd-df5ca7b08060"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:34:28.497721 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.497680 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xqqt\" (UniqueName: \"kubernetes.io/projected/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-kube-api-access-5xqqt\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:34:28.497721 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.497713 2541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istiod-ca-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:34:28.497721 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.497723 2541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-envoy\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:34:28.497721 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.497730 2541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-token\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:34:28.498006 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.497739 2541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-data\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:34:28.498006 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:28.497746 2541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060-istio-podinfo\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:34:29.223380 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:29.223340 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" event={"ID":"00f0a5c3-cd3a-4186-a3fd-df5ca7b08060","Type":"ContainerDied","Data":"071ccab4f87db1d81f3be4fbd892e6e91bd1de42d8c766c180eaf0060caa8c54"} Sep 10 16:34:29.223853 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:29.223411 2541 scope.go:117] "RemoveContainer" containerID="6a3b6951f686f4ee14d636396c0909146e01f613fa2c79fa87f10620747a31c3" Sep 10 16:34:29.223853 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:29.223350 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk" Sep 10 16:34:29.242963 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:29.242869 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk"] Sep 10 16:34:29.247766 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:29.247726 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-275a06-openshift-default-5c4b879898-mjctk"] Sep 10 16:34:30.981141 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:30.981098 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" path="/var/lib/kubelet/pods/00f0a5c3-cd3a-4186-a3fd-df5ca7b08060/volumes" Sep 10 16:34:33.870497 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:33.870400 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:33.872576 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:33.872540 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-219fd6-29817630-sp7jf"] Sep 10 16:34:33.875086 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:33.875055 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-219fd6-29817630-sp7jf"] Sep 10 16:34:33.974607 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:33.974564 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:34.189740 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:34.189700 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:34.273863 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:34.273826 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:34.557979 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:34.557880 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:34.980135 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:34.980089 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="515fda7a-e2bb-4542-a0f1-71f019b374fd" path="/var/lib/kubelet/pods/515fda7a-e2bb-4542-a0f1-71f019b374fd/volumes" Sep 10 16:34:37.454897 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:37.454851 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:34:41.259431 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:41.259384 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:41.369055 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:41.369016 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:41.543945 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:41.543845 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f"] Sep 10 16:34:41.544116 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:41.544092 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" podUID="74ab292d-0342-4278-943b-a85c78d351b9" containerName="istio-proxy" containerID="cri-o://e34dbff76f2ff42e895050d4d0df9684660632b5f5d07f45a948d3e11561b17c" gracePeriod=30 Sep 10 16:34:41.558732 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:41.558696 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:42.069489 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:42.069452 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:46.807488 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.807448 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:34:46.856683 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.856587 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-workload-certs\") pod \"74ab292d-0342-4278-943b-a85c78d351b9\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " Sep 10 16:34:46.856683 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.856628 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m22ks\" (UniqueName: \"kubernetes.io/projected/74ab292d-0342-4278-943b-a85c78d351b9-kube-api-access-m22ks\") pod \"74ab292d-0342-4278-943b-a85c78d351b9\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " Sep 10 16:34:46.856683 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.856650 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ab292d-0342-4278-943b-a85c78d351b9-istiod-ca-cert\") pod \"74ab292d-0342-4278-943b-a85c78d351b9\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " Sep 10 16:34:46.856683 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.856684 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ab292d-0342-4278-943b-a85c78d351b9-istio-token\") pod \"74ab292d-0342-4278-943b-a85c78d351b9\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " Sep 10 16:34:46.857020 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.856731 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-credential-socket\") pod \"74ab292d-0342-4278-943b-a85c78d351b9\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " Sep 10 16:34:46.857020 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.856760 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-istio-data\") pod \"74ab292d-0342-4278-943b-a85c78d351b9\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " Sep 10 16:34:46.857020 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.856948 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "74ab292d-0342-4278-943b-a85c78d351b9" (UID: "74ab292d-0342-4278-943b-a85c78d351b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:46.857181 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.857108 2541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-workload-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:34:46.857181 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.857118 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-istio-data" (OuterVolumeSpecName: "istio-data") pod "74ab292d-0342-4278-943b-a85c78d351b9" (UID: "74ab292d-0342-4278-943b-a85c78d351b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:46.857322 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.857182 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "74ab292d-0342-4278-943b-a85c78d351b9" (UID: "74ab292d-0342-4278-943b-a85c78d351b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:46.857322 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.857239 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ab292d-0342-4278-943b-a85c78d351b9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "74ab292d-0342-4278-943b-a85c78d351b9" (UID: "74ab292d-0342-4278-943b-a85c78d351b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:34:46.859412 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.859375 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ab292d-0342-4278-943b-a85c78d351b9-kube-api-access-m22ks" (OuterVolumeSpecName: "kube-api-access-m22ks") pod "74ab292d-0342-4278-943b-a85c78d351b9" (UID: "74ab292d-0342-4278-943b-a85c78d351b9"). InnerVolumeSpecName "kube-api-access-m22ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:34:46.859540 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.859469 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ab292d-0342-4278-943b-a85c78d351b9-istio-token" (OuterVolumeSpecName: "istio-token") pod "74ab292d-0342-4278-943b-a85c78d351b9" (UID: "74ab292d-0342-4278-943b-a85c78d351b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:34:46.957888 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.957832 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-workload-socket\") pod \"74ab292d-0342-4278-943b-a85c78d351b9\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " Sep 10 16:34:46.957888 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.957890 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ab292d-0342-4278-943b-a85c78d351b9-istio-podinfo\") pod \"74ab292d-0342-4278-943b-a85c78d351b9\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " Sep 10 16:34:46.958165 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.957927 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-istio-envoy\") pod \"74ab292d-0342-4278-943b-a85c78d351b9\" (UID: \"74ab292d-0342-4278-943b-a85c78d351b9\") " Sep 10 16:34:46.958165 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.958083 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m22ks\" (UniqueName: \"kubernetes.io/projected/74ab292d-0342-4278-943b-a85c78d351b9-kube-api-access-m22ks\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:34:46.958165 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.958094 2541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ab292d-0342-4278-943b-a85c78d351b9-istiod-ca-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:34:46.958165 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.958103 2541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ab292d-0342-4278-943b-a85c78d351b9-istio-token\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:34:46.958165 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.958111 2541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-credential-socket\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:34:46.958165 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.958118 2541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-istio-data\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:34:46.958415 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.958164 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "74ab292d-0342-4278-943b-a85c78d351b9" (UID: "74ab292d-0342-4278-943b-a85c78d351b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:46.960398 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.960367 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74ab292d-0342-4278-943b-a85c78d351b9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "74ab292d-0342-4278-943b-a85c78d351b9" (UID: "74ab292d-0342-4278-943b-a85c78d351b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:34:46.960476 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:46.960400 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "74ab292d-0342-4278-943b-a85c78d351b9" (UID: "74ab292d-0342-4278-943b-a85c78d351b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:47.058985 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:47.058939 2541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-workload-socket\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:34:47.058985 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:47.058978 2541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ab292d-0342-4278-943b-a85c78d351b9-istio-podinfo\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:34:47.058985 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:47.058988 2541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ab292d-0342-4278-943b-a85c78d351b9-istio-envoy\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:34:47.353515 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:47.353476 2541 generic.go:356] "Generic (PLEG): container finished" podID="74ab292d-0342-4278-943b-a85c78d351b9" containerID="e34dbff76f2ff42e895050d4d0df9684660632b5f5d07f45a948d3e11561b17c" exitCode=0 Sep 10 16:34:47.353704 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:47.353534 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" Sep 10 16:34:47.353704 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:47.353571 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" event={"ID":"74ab292d-0342-4278-943b-a85c78d351b9","Type":"ContainerDied","Data":"e34dbff76f2ff42e895050d4d0df9684660632b5f5d07f45a948d3e11561b17c"} Sep 10 16:34:47.353704 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:47.353637 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f" event={"ID":"74ab292d-0342-4278-943b-a85c78d351b9","Type":"ContainerDied","Data":"b38e06de8a1726c8460815b890e65c094c9289cc6ea6582c1187a7d89fe5419c"} Sep 10 16:34:47.353704 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:47.353665 2541 scope.go:117] "RemoveContainer" containerID="e34dbff76f2ff42e895050d4d0df9684660632b5f5d07f45a948d3e11561b17c" Sep 10 16:34:47.364424 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:47.364394 2541 scope.go:117] "RemoveContainer" containerID="f0af5e19adbee84bfda95c6f76140eb2a99b32dc85dc849e6e1234245b626134" Sep 10 16:34:47.374191 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:47.374168 2541 scope.go:117] "RemoveContainer" containerID="e34dbff76f2ff42e895050d4d0df9684660632b5f5d07f45a948d3e11561b17c" Sep 10 16:34:47.374652 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:34:47.374626 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e34dbff76f2ff42e895050d4d0df9684660632b5f5d07f45a948d3e11561b17c\": container with ID starting with e34dbff76f2ff42e895050d4d0df9684660632b5f5d07f45a948d3e11561b17c not found: ID does not exist" containerID="e34dbff76f2ff42e895050d4d0df9684660632b5f5d07f45a948d3e11561b17c" Sep 10 16:34:47.374736 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:47.374663 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e34dbff76f2ff42e895050d4d0df9684660632b5f5d07f45a948d3e11561b17c"} err="failed to get container status \"e34dbff76f2ff42e895050d4d0df9684660632b5f5d07f45a948d3e11561b17c\": rpc error: code = NotFound desc = could not find container \"e34dbff76f2ff42e895050d4d0df9684660632b5f5d07f45a948d3e11561b17c\": container with ID starting with e34dbff76f2ff42e895050d4d0df9684660632b5f5d07f45a948d3e11561b17c not found: ID does not exist" Sep 10 16:34:47.374736 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:47.374685 2541 scope.go:117] "RemoveContainer" containerID="f0af5e19adbee84bfda95c6f76140eb2a99b32dc85dc849e6e1234245b626134" Sep 10 16:34:47.374991 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:34:47.374973 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0af5e19adbee84bfda95c6f76140eb2a99b32dc85dc849e6e1234245b626134\": container with ID starting with f0af5e19adbee84bfda95c6f76140eb2a99b32dc85dc849e6e1234245b626134 not found: ID does not exist" containerID="f0af5e19adbee84bfda95c6f76140eb2a99b32dc85dc849e6e1234245b626134" Sep 10 16:34:47.375048 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:47.374995 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0af5e19adbee84bfda95c6f76140eb2a99b32dc85dc849e6e1234245b626134"} err="failed to get container status \"f0af5e19adbee84bfda95c6f76140eb2a99b32dc85dc849e6e1234245b626134\": rpc error: code = NotFound desc = could not find container \"f0af5e19adbee84bfda95c6f76140eb2a99b32dc85dc849e6e1234245b626134\": container with ID starting with f0af5e19adbee84bfda95c6f76140eb2a99b32dc85dc849e6e1234245b626134 not found: ID does not exist" Sep 10 16:34:47.379634 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:47.379597 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f"] Sep 10 16:34:47.386467 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:47.386430 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-219fd6-openshift-default-56dc8b44d8-tkv4f"] Sep 10 16:34:48.979629 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:48.979581 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74ab292d-0342-4278-943b-a85c78d351b9" path="/var/lib/kubelet/pods/74ab292d-0342-4278-943b-a85c78d351b9/volumes" Sep 10 16:34:52.637590 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:52.637547 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b73303-29817630-84psd"] Sep 10 16:34:52.642232 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:52.642195 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b73303-29817630-84psd"] Sep 10 16:34:52.647582 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:52.647529 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:52.746375 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:52.746327 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:52.886834 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:52.886796 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:52.966436 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:52.966352 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:52.981394 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:52.981360 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86e54c8c-d5e2-430d-bce5-7596c8fd9bc0" path="/var/lib/kubelet/pods/86e54c8c-d5e2-430d-bce5-7596c8fd9bc0/volumes" Sep 10 16:34:53.319562 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:53.319475 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:55.248761 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:34:55.248726 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:35:00.088190 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:00.088149 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:00.186332 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:00.186272 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:00.341517 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:00.341404 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp"] Sep 10 16:35:00.341709 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:00.341660 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" podUID="a0e1c048-296e-4512-98fe-085cc50d61b6" containerName="istio-proxy" containerID="cri-o://b177a2345535b2a3ca4eadf863c9f867955da4da8da792b06cdd6e2bf33769df" gracePeriod=30 Sep 10 16:35:00.374284 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:00.374253 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:04.766681 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:04.766637 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:35:04.878997 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:04.878963 2541 patch_prober.go:28] interesting pod/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 16:35:04.879180 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:04.879051 2541 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" podUID="a0e1c048-296e-4512-98fe-085cc50d61b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 16:35:05.477229 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.477199 2541 generic.go:356] "Generic (PLEG): container finished" podID="a0e1c048-296e-4512-98fe-085cc50d61b6" containerID="b177a2345535b2a3ca4eadf863c9f867955da4da8da792b06cdd6e2bf33769df" exitCode=0 Sep 10 16:35:05.477229 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.477241 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" event={"ID":"a0e1c048-296e-4512-98fe-085cc50d61b6","Type":"ContainerDied","Data":"b177a2345535b2a3ca4eadf863c9f867955da4da8da792b06cdd6e2bf33769df"} Sep 10 16:35:05.477568 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.477320 2541 scope.go:117] "RemoveContainer" containerID="d690a44ffc80a1951d24d671c594c622945dd661cdc73bcb63cbf95569280a3b" Sep 10 16:35:05.612673 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.612593 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:35:05.729644 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.729605 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-envoy\") pod \"a0e1c048-296e-4512-98fe-085cc50d61b6\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " Sep 10 16:35:05.729644 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.729645 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-credential-socket\") pod \"a0e1c048-296e-4512-98fe-085cc50d61b6\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " Sep 10 16:35:05.729924 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.729684 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvkpb\" (UniqueName: \"kubernetes.io/projected/a0e1c048-296e-4512-98fe-085cc50d61b6-kube-api-access-tvkpb\") pod \"a0e1c048-296e-4512-98fe-085cc50d61b6\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " Sep 10 16:35:05.729924 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.729725 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-data\") pod \"a0e1c048-296e-4512-98fe-085cc50d61b6\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " Sep 10 16:35:05.729924 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.729758 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-workload-certs\") pod \"a0e1c048-296e-4512-98fe-085cc50d61b6\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " Sep 10 16:35:05.729924 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.729801 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e1c048-296e-4512-98fe-085cc50d61b6-istiod-ca-cert\") pod \"a0e1c048-296e-4512-98fe-085cc50d61b6\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " Sep 10 16:35:05.729924 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.729848 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-podinfo\") pod \"a0e1c048-296e-4512-98fe-085cc50d61b6\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " Sep 10 16:35:05.729924 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.729878 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-workload-socket\") pod \"a0e1c048-296e-4512-98fe-085cc50d61b6\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " Sep 10 16:35:05.730243 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.730114 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-data" (OuterVolumeSpecName: "istio-data") pod "a0e1c048-296e-4512-98fe-085cc50d61b6" (UID: "a0e1c048-296e-4512-98fe-085cc50d61b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:35:05.730243 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.730158 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a0e1c048-296e-4512-98fe-085cc50d61b6" (UID: "a0e1c048-296e-4512-98fe-085cc50d61b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:35:05.730243 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.730175 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-token\") pod \"a0e1c048-296e-4512-98fe-085cc50d61b6\" (UID: \"a0e1c048-296e-4512-98fe-085cc50d61b6\") " Sep 10 16:35:05.730497 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.730471 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a0e1c048-296e-4512-98fe-085cc50d61b6" (UID: "a0e1c048-296e-4512-98fe-085cc50d61b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:35:05.730604 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.730565 2541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-data\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:35:05.730716 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.730692 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a0e1c048-296e-4512-98fe-085cc50d61b6" (UID: "a0e1c048-296e-4512-98fe-085cc50d61b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:35:05.730949 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.730775 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e1c048-296e-4512-98fe-085cc50d61b6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a0e1c048-296e-4512-98fe-085cc50d61b6" (UID: "a0e1c048-296e-4512-98fe-085cc50d61b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:35:05.731060 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.731036 2541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-workload-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:35:05.732257 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.732232 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a0e1c048-296e-4512-98fe-085cc50d61b6" (UID: "a0e1c048-296e-4512-98fe-085cc50d61b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:35:05.732407 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.732359 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a0e1c048-296e-4512-98fe-085cc50d61b6" (UID: "a0e1c048-296e-4512-98fe-085cc50d61b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:35:05.732822 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.732801 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-token" (OuterVolumeSpecName: "istio-token") pod "a0e1c048-296e-4512-98fe-085cc50d61b6" (UID: "a0e1c048-296e-4512-98fe-085cc50d61b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:35:05.732919 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.732825 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e1c048-296e-4512-98fe-085cc50d61b6-kube-api-access-tvkpb" (OuterVolumeSpecName: "kube-api-access-tvkpb") pod "a0e1c048-296e-4512-98fe-085cc50d61b6" (UID: "a0e1c048-296e-4512-98fe-085cc50d61b6"). InnerVolumeSpecName "kube-api-access-tvkpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:35:05.831696 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.831651 2541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-podinfo\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:35:05.831696 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.831686 2541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-workload-socket\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:35:05.831696 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.831696 2541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-token\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:35:05.831696 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.831705 2541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-istio-envoy\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:35:05.832206 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.831714 2541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e1c048-296e-4512-98fe-085cc50d61b6-credential-socket\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:35:05.832206 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.831724 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvkpb\" (UniqueName: \"kubernetes.io/projected/a0e1c048-296e-4512-98fe-085cc50d61b6-kube-api-access-tvkpb\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:35:05.832206 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:05.831733 2541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e1c048-296e-4512-98fe-085cc50d61b6-istiod-ca-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:35:06.486951 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:06.486918 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" event={"ID":"a0e1c048-296e-4512-98fe-085cc50d61b6","Type":"ContainerDied","Data":"a2bffbb80c3decd988dcbc3458f80fd45cafa953dd5e737b9e704b3853e7af74"} Sep 10 16:35:06.486951 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:06.486949 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp" Sep 10 16:35:06.487200 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:06.486980 2541 scope.go:117] "RemoveContainer" containerID="b177a2345535b2a3ca4eadf863c9f867955da4da8da792b06cdd6e2bf33769df" Sep 10 16:35:06.510533 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:06.510496 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp"] Sep 10 16:35:06.515934 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:06.515890 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b73303-openshift-default-68688bdd7d-vfncp"] Sep 10 16:35:06.980887 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:35:06.980835 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0e1c048-296e-4512-98fe-085cc50d61b6" path="/var/lib/kubelet/pods/a0e1c048-296e-4512-98fe-085cc50d61b6/volumes" Sep 10 16:36:23.798698 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:36:23.798657 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:36:25.065602 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:36:25.065566 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:36:25.487240 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:36:25.487206 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:36:26.062647 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:36:26.062604 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:36:27.397238 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:36:27.397203 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:36:41.184592 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:36:41.184555 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/1.log" Sep 10 16:36:41.188142 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:36:41.188113 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/1.log" Sep 10 16:36:41.190885 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:36:41.190858 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4c6j_2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd/ovn-acl-logging/0.log" Sep 10 16:36:41.194412 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:36:41.194385 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4c6j_2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd/ovn-acl-logging/0.log" Sep 10 16:36:41.415451 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:36:41.415411 2541 scope.go:117] "RemoveContainer" containerID="7f4c8327e16cb1cf29e0a477505bacd3590925914f1cbc6be2491082a1605b0f" Sep 10 16:36:41.425563 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:36:41.425536 2541 scope.go:117] "RemoveContainer" containerID="613bbaca2988075963ddefc339b8258c282107abcd358a8d2de2cce12d72ae00" Sep 10 16:36:41.435357 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:36:41.435329 2541 scope.go:117] "RemoveContainer" containerID="5124aced9dd0cd8cebcbe05989359f70187d5f4f6c462ff4dea110f45da3c092" Sep 10 16:36:41.445208 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:36:41.445185 2541 scope.go:117] "RemoveContainer" containerID="52f09693e7b206cb3c150febed62648b805797f8493f3198337cd416cdc304ea" Sep 10 16:36:41.455205 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:36:41.455180 2541 scope.go:117] "RemoveContainer" containerID="f35fb5725170e48f38c7c284a88658e9f4b473755221fd55366e6018d4b87d2c" Sep 10 16:36:52.756829 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:36:52.756781 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:37:00.962897 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:00.962851 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:37:07.800828 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:07.800776 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:37:14.067570 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:14.067529 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:37:26.166343 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:26.166286 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:37:32.550944 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:32.550854 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:37:41.497764 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:41.497729 2541 scope.go:117] "RemoveContainer" containerID="a96bba79d30186e7840c95efee33c5d53b6163ce46803dee175cf5f1c60af81d" Sep 10 16:37:41.507354 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:41.507325 2541 scope.go:117] "RemoveContainer" containerID="a3064bd9b9151398c7563960e28068e28aa56097fa2893068741697b445b343f" Sep 10 16:37:46.920890 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:46.920841 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6"] Sep 10 16:37:46.921374 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:46.921087 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6" podUID="b8d2baaf-45b6-4250-9b68-6653285e8951" containerName="manager" containerID="cri-o://0d62b080aeaeb9a8c28395021e1f093a5c146ba61272ac47ef75ea71f44e50aa" gracePeriod=10 Sep 10 16:37:47.175748 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:47.175653 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6" Sep 10 16:37:47.245592 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:47.245556 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbnxx\" (UniqueName: \"kubernetes.io/projected/b8d2baaf-45b6-4250-9b68-6653285e8951-kube-api-access-gbnxx\") pod \"b8d2baaf-45b6-4250-9b68-6653285e8951\" (UID: \"b8d2baaf-45b6-4250-9b68-6653285e8951\") " Sep 10 16:37:47.245766 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:47.245624 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8d2baaf-45b6-4250-9b68-6653285e8951-extensions-socket-volume\") pod \"b8d2baaf-45b6-4250-9b68-6653285e8951\" (UID: \"b8d2baaf-45b6-4250-9b68-6653285e8951\") " Sep 10 16:37:47.246029 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:47.245999 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d2baaf-45b6-4250-9b68-6653285e8951-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b8d2baaf-45b6-4250-9b68-6653285e8951" (UID: "b8d2baaf-45b6-4250-9b68-6653285e8951"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:37:47.247950 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:47.247919 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d2baaf-45b6-4250-9b68-6653285e8951-kube-api-access-gbnxx" (OuterVolumeSpecName: "kube-api-access-gbnxx") pod "b8d2baaf-45b6-4250-9b68-6653285e8951" (UID: "b8d2baaf-45b6-4250-9b68-6653285e8951"). InnerVolumeSpecName "kube-api-access-gbnxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:37:47.346589 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:47.346548 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbnxx\" (UniqueName: \"kubernetes.io/projected/b8d2baaf-45b6-4250-9b68-6653285e8951-kube-api-access-gbnxx\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:37:47.346589 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:47.346579 2541 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8d2baaf-45b6-4250-9b68-6653285e8951-extensions-socket-volume\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:37:47.489125 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:47.489029 2541 generic.go:356] "Generic (PLEG): container finished" podID="b8d2baaf-45b6-4250-9b68-6653285e8951" containerID="0d62b080aeaeb9a8c28395021e1f093a5c146ba61272ac47ef75ea71f44e50aa" exitCode=0 Sep 10 16:37:47.489125 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:47.489074 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6" Sep 10 16:37:47.489125 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:47.489094 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6" event={"ID":"b8d2baaf-45b6-4250-9b68-6653285e8951","Type":"ContainerDied","Data":"0d62b080aeaeb9a8c28395021e1f093a5c146ba61272ac47ef75ea71f44e50aa"} Sep 10 16:37:47.489406 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:47.489151 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6" event={"ID":"b8d2baaf-45b6-4250-9b68-6653285e8951","Type":"ContainerDied","Data":"ff8e1d6a7d4ffb3c57cce959bd718a05ffea47c31a002a1efe0a7b12ec89beb4"} Sep 10 16:37:47.489406 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:47.489180 2541 scope.go:117] "RemoveContainer" containerID="0d62b080aeaeb9a8c28395021e1f093a5c146ba61272ac47ef75ea71f44e50aa" Sep 10 16:37:47.503209 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:47.503188 2541 scope.go:117] "RemoveContainer" containerID="0d62b080aeaeb9a8c28395021e1f093a5c146ba61272ac47ef75ea71f44e50aa" Sep 10 16:37:47.503632 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:37:47.503607 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d62b080aeaeb9a8c28395021e1f093a5c146ba61272ac47ef75ea71f44e50aa\": container with ID starting with 0d62b080aeaeb9a8c28395021e1f093a5c146ba61272ac47ef75ea71f44e50aa not found: ID does not exist" containerID="0d62b080aeaeb9a8c28395021e1f093a5c146ba61272ac47ef75ea71f44e50aa" Sep 10 16:37:47.503713 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:47.503643 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d62b080aeaeb9a8c28395021e1f093a5c146ba61272ac47ef75ea71f44e50aa"} err="failed to get container status \"0d62b080aeaeb9a8c28395021e1f093a5c146ba61272ac47ef75ea71f44e50aa\": rpc error: code = NotFound desc = could not find container \"0d62b080aeaeb9a8c28395021e1f093a5c146ba61272ac47ef75ea71f44e50aa\": container with ID starting with 0d62b080aeaeb9a8c28395021e1f093a5c146ba61272ac47ef75ea71f44e50aa not found: ID does not exist" Sep 10 16:37:47.518973 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:47.518922 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6"] Sep 10 16:37:47.541557 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:47.541511 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-jwzb6"] Sep 10 16:37:48.979649 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:37:48.979618 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8d2baaf-45b6-4250-9b68-6653285e8951" path="/var/lib/kubelet/pods/b8d2baaf-45b6-4250-9b68-6653285e8951/volumes" Sep 10 16:38:47.649529 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.649479 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98rc4"] Sep 10 16:38:47.650179 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.650042 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e1c048-296e-4512-98fe-085cc50d61b6" containerName="istio-proxy" Sep 10 16:38:47.650179 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.650064 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e1c048-296e-4512-98fe-085cc50d61b6" containerName="istio-proxy" Sep 10 16:38:47.650179 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.650085 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" containerName="istio-proxy" Sep 10 16:38:47.650179 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.650095 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" containerName="istio-proxy" Sep 10 16:38:47.650179 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.650106 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2" containerName="revoke-keys" Sep 10 16:38:47.650179 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.650115 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2" containerName="revoke-keys" Sep 10 16:38:47.650179 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.650131 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8d2baaf-45b6-4250-9b68-6653285e8951" containerName="manager" Sep 10 16:38:47.650179 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.650140 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8d2baaf-45b6-4250-9b68-6653285e8951" containerName="manager" Sep 10 16:38:47.650179 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.650157 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74ab292d-0342-4278-943b-a85c78d351b9" containerName="istio-proxy" Sep 10 16:38:47.650179 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.650165 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="74ab292d-0342-4278-943b-a85c78d351b9" containerName="istio-proxy" Sep 10 16:38:47.650179 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.650176 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74ab292d-0342-4278-943b-a85c78d351b9" containerName="istio-proxy" Sep 10 16:38:47.650179 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.650184 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="74ab292d-0342-4278-943b-a85c78d351b9" containerName="istio-proxy" Sep 10 16:38:47.650801 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.650200 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bad02186-02e9-4565-9c18-34a59d63a36e" containerName="payload-pre-processing" Sep 10 16:38:47.650801 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.650208 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="bad02186-02e9-4565-9c18-34a59d63a36e" containerName="payload-pre-processing" Sep 10 16:38:47.650801 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.650291 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="74ab292d-0342-4278-943b-a85c78d351b9" containerName="istio-proxy" Sep 10 16:38:47.650801 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.650330 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0e1c048-296e-4512-98fe-085cc50d61b6" containerName="istio-proxy" Sep 10 16:38:47.650801 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.650342 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8d2baaf-45b6-4250-9b68-6653285e8951" containerName="manager" Sep 10 16:38:47.650801 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.650351 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf7a7a1-5ac6-4e7c-9474-11d6444fc0a2" containerName="revoke-keys" Sep 10 16:38:47.650801 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.650359 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" containerName="istio-proxy" Sep 10 16:38:47.650801 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.650369 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="bad02186-02e9-4565-9c18-34a59d63a36e" containerName="payload-pre-processing" Sep 10 16:38:47.650801 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.650378 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" containerName="istio-proxy" Sep 10 16:38:47.653757 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.653731 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98rc4" Sep 10 16:38:47.656019 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.655992 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n4rkc\"" Sep 10 16:38:47.689654 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.689607 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98rc4"] Sep 10 16:38:47.780662 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.780622 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw9f2\" (UniqueName: \"kubernetes.io/projected/bc3a0463-7280-48ed-b8a7-38f5cd64b2d9-kube-api-access-nw9f2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-98rc4\" (UID: \"bc3a0463-7280-48ed-b8a7-38f5cd64b2d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98rc4" Sep 10 16:38:47.780846 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.780693 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc3a0463-7280-48ed-b8a7-38f5cd64b2d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-98rc4\" (UID: \"bc3a0463-7280-48ed-b8a7-38f5cd64b2d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98rc4" Sep 10 16:38:47.881243 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.881205 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc3a0463-7280-48ed-b8a7-38f5cd64b2d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-98rc4\" (UID: \"bc3a0463-7280-48ed-b8a7-38f5cd64b2d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98rc4" Sep 10 16:38:47.881491 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.881268 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw9f2\" (UniqueName: \"kubernetes.io/projected/bc3a0463-7280-48ed-b8a7-38f5cd64b2d9-kube-api-access-nw9f2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-98rc4\" (UID: \"bc3a0463-7280-48ed-b8a7-38f5cd64b2d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98rc4" Sep 10 16:38:47.881692 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.881664 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc3a0463-7280-48ed-b8a7-38f5cd64b2d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-98rc4\" (UID: \"bc3a0463-7280-48ed-b8a7-38f5cd64b2d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98rc4" Sep 10 16:38:47.890592 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.890555 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw9f2\" (UniqueName: \"kubernetes.io/projected/bc3a0463-7280-48ed-b8a7-38f5cd64b2d9-kube-api-access-nw9f2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-98rc4\" (UID: \"bc3a0463-7280-48ed-b8a7-38f5cd64b2d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98rc4" Sep 10 16:38:47.964572 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:47.964467 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98rc4" Sep 10 16:38:48.259842 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:48.259816 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98rc4"] Sep 10 16:38:48.261240 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:38:48.261214 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3a0463_7280_48ed_b8a7_38f5cd64b2d9.slice/crio-22276c0df625e0cab79d8c0aa88e6c6b11332f800cbe2c8e10b6d19835488153 WatchSource:0}: Error finding container 22276c0df625e0cab79d8c0aa88e6c6b11332f800cbe2c8e10b6d19835488153: Status 404 returned error can't find the container with id 22276c0df625e0cab79d8c0aa88e6c6b11332f800cbe2c8e10b6d19835488153 Sep 10 16:38:48.848212 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:48.848176 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98rc4" event={"ID":"bc3a0463-7280-48ed-b8a7-38f5cd64b2d9","Type":"ContainerStarted","Data":"f7f2b784a493137bcac7b85e20b00288e0e795c866e43e65586b06dd0d49e00b"} Sep 10 16:38:48.848671 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:48.848232 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98rc4" event={"ID":"bc3a0463-7280-48ed-b8a7-38f5cd64b2d9","Type":"ContainerStarted","Data":"22276c0df625e0cab79d8c0aa88e6c6b11332f800cbe2c8e10b6d19835488153"} Sep 10 16:38:48.848671 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:48.848292 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98rc4" Sep 10 16:38:48.883831 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:48.883769 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98rc4" podStartSLOduration=1.883749694 podStartE2EDuration="1.883749694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:38:48.8741757 +0000 UTC m=+2228.397420894" watchObservedRunningTime="2026-09-10 16:38:48.883749694 +0000 UTC m=+2228.406994892" Sep 10 16:38:59.856183 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:38:59.856145 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98rc4" Sep 10 16:39:07.615582 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:07.615491 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:10.065000 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:10.064929 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:39:18.563168 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:18.563129 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:39:27.350483 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:27.350441 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:39:32.940839 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:32.940803 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:35.760912 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:35.760867 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:39:41.549425 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:41.549387 2541 scope.go:117] "RemoveContainer" containerID="fb4d5f235da97f054fa0e72dd01e0d78171207cd10860120f26c8b4e337ac2f7" Sep 10 16:39:41.559490 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:41.559466 2541 scope.go:117] "RemoveContainer" containerID="5ed8ef20680e2d11f2412739bd716375e0e75b633a2325e359b7150cce361806" Sep 10 16:39:56.940534 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:56.940490 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp"] Sep 10 16:39:56.940957 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:56.940895 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e1c048-296e-4512-98fe-085cc50d61b6" containerName="istio-proxy" Sep 10 16:39:56.940957 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:56.940906 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e1c048-296e-4512-98fe-085cc50d61b6" containerName="istio-proxy" Sep 10 16:39:56.940957 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:56.940920 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" containerName="istio-proxy" Sep 10 16:39:56.940957 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:56.940926 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="00f0a5c3-cd3a-4186-a3fd-df5ca7b08060" containerName="istio-proxy" Sep 10 16:39:56.941100 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:56.940992 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="74ab292d-0342-4278-943b-a85c78d351b9" containerName="istio-proxy" Sep 10 16:39:56.941100 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:56.941002 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0e1c048-296e-4512-98fe-085cc50d61b6" containerName="istio-proxy" Sep 10 16:39:56.944031 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:56.944009 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:39:56.946543 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:56.946521 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-1d0deeff-kserve-self-signed-certs\"" Sep 10 16:39:56.956932 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:56.956896 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp"] Sep 10 16:39:56.998856 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:56.998821 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:57.027904 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:57.027859 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-model-cache\") pod \"e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:39:57.027904 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:57.027910 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgpcr\" (UniqueName: \"kubernetes.io/projected/a43ee067-34a1-448d-ab0f-7e990af56058-kube-api-access-fgpcr\") pod \"e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:39:57.028155 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:57.027984 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-dshm\") pod \"e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:39:57.028155 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:57.028043 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-home\") pod \"e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:39:57.028155 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:57.028098 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-tmp-dir\") pod \"e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:39:57.028155 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:57.028136 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43ee067-34a1-448d-ab0f-7e990af56058-tls-certs\") pod \"e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:39:57.128707 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:57.128661 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43ee067-34a1-448d-ab0f-7e990af56058-tls-certs\") pod \"e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:39:57.128707 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:57.128712 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-model-cache\") pod \"e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:39:57.128961 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:57.128731 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgpcr\" (UniqueName: \"kubernetes.io/projected/a43ee067-34a1-448d-ab0f-7e990af56058-kube-api-access-fgpcr\") pod \"e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:39:57.128961 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:57.128771 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-dshm\") pod \"e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:39:57.128961 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:57.128797 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-home\") pod \"e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:39:57.128961 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:57.128826 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-tmp-dir\") pod \"e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:39:57.129191 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:57.129173 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-home\") pod \"e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:39:57.129254 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:57.129173 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-model-cache\") pod \"e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:39:57.129254 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:57.129219 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-tmp-dir\") pod \"e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:39:57.131518 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:57.131484 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-dshm\") pod \"e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:39:57.131708 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:57.131686 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43ee067-34a1-448d-ab0f-7e990af56058-tls-certs\") pod \"e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:39:57.140733 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:57.140700 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgpcr\" (UniqueName: \"kubernetes.io/projected/a43ee067-34a1-448d-ab0f-7e990af56058-kube-api-access-fgpcr\") pod \"e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:39:57.253645 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:57.253546 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:39:57.564726 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:57.564694 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp"] Sep 10 16:39:57.566796 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:39:57.566768 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda43ee067_34a1_448d_ab0f_7e990af56058.slice/crio-e29e86c118b34f9c0e36f8985b2e121e06277f57b642874e4b255c3008110519 WatchSource:0}: Error finding container e29e86c118b34f9c0e36f8985b2e121e06277f57b642874e4b255c3008110519: Status 404 returned error can't find the container with id e29e86c118b34f9c0e36f8985b2e121e06277f57b642874e4b255c3008110519 Sep 10 16:39:58.264587 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:58.264541 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" event={"ID":"a43ee067-34a1-448d-ab0f-7e990af56058","Type":"ContainerStarted","Data":"7b0f10984abd6ec26d705f3986ac0622e5ef9261a8bc69932b43dfb2d608e8dd"} Sep 10 16:39:58.265007 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:58.264596 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" event={"ID":"a43ee067-34a1-448d-ab0f-7e990af56058","Type":"ContainerStarted","Data":"e29e86c118b34f9c0e36f8985b2e121e06277f57b642874e4b255c3008110519"} Sep 10 16:39:58.295035 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:39:58.294980 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" podStartSLOduration=2.294961613 podStartE2EDuration="2.294961613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:39:58.29213853 +0000 UTC m=+2297.815383958" watchObservedRunningTime="2026-09-10 16:39:58.294961613 +0000 UTC m=+2297.818206806" Sep 10 16:40:00.088121 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:00.088058 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:40:07.254180 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:07.254137 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:40:07.254180 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:07.254193 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:40:07.267253 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:07.267211 2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:40:07.330854 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:07.330821 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:40:15.645194 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:15.645154 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:18.166338 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:18.166278 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp"] Sep 10 16:40:18.226390 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:18.226352 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:18.410322 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:18.410264 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:40:26.614031 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:26.613991 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:40:31.122070 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:31.122033 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:33.187708 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.187598 2541 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" podUID="a43ee067-34a1-448d-ab0f-7e990af56058" containerName="main" containerID="cri-o://7b0f10984abd6ec26d705f3986ac0622e5ef9261a8bc69932b43dfb2d608e8dd" gracePeriod=45 Sep 10 16:40:33.452440 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.452369 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:40:33.469879 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.469839 2541 generic.go:356] "Generic (PLEG): container finished" podID="a43ee067-34a1-448d-ab0f-7e990af56058" containerID="7b0f10984abd6ec26d705f3986ac0622e5ef9261a8bc69932b43dfb2d608e8dd" exitCode=0 Sep 10 16:40:33.470054 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.469888 2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" Sep 10 16:40:33.470054 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.469917 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" event={"ID":"a43ee067-34a1-448d-ab0f-7e990af56058","Type":"ContainerDied","Data":"7b0f10984abd6ec26d705f3986ac0622e5ef9261a8bc69932b43dfb2d608e8dd"} Sep 10 16:40:33.470054 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.469963 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp" event={"ID":"a43ee067-34a1-448d-ab0f-7e990af56058","Type":"ContainerDied","Data":"e29e86c118b34f9c0e36f8985b2e121e06277f57b642874e4b255c3008110519"} Sep 10 16:40:33.470054 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.469982 2541 scope.go:117] "RemoveContainer" containerID="7b0f10984abd6ec26d705f3986ac0622e5ef9261a8bc69932b43dfb2d608e8dd" Sep 10 16:40:33.479935 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.479911 2541 scope.go:117] "RemoveContainer" containerID="7b0f10984abd6ec26d705f3986ac0622e5ef9261a8bc69932b43dfb2d608e8dd" Sep 10 16:40:33.480359 ip-10-0-137-172 kubenswrapper[2541]: E0910 16:40:33.480336 2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b0f10984abd6ec26d705f3986ac0622e5ef9261a8bc69932b43dfb2d608e8dd\": container with ID starting with 7b0f10984abd6ec26d705f3986ac0622e5ef9261a8bc69932b43dfb2d608e8dd not found: ID does not exist" containerID="7b0f10984abd6ec26d705f3986ac0622e5ef9261a8bc69932b43dfb2d608e8dd" Sep 10 16:40:33.480458 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.480370 2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b0f10984abd6ec26d705f3986ac0622e5ef9261a8bc69932b43dfb2d608e8dd"} err="failed to get container status \"7b0f10984abd6ec26d705f3986ac0622e5ef9261a8bc69932b43dfb2d608e8dd\": rpc error: code = NotFound desc = could not find container \"7b0f10984abd6ec26d705f3986ac0622e5ef9261a8bc69932b43dfb2d608e8dd\": container with ID starting with 7b0f10984abd6ec26d705f3986ac0622e5ef9261a8bc69932b43dfb2d608e8dd not found: ID does not exist" Sep 10 16:40:33.576945 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.576906 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-model-cache\") pod \"a43ee067-34a1-448d-ab0f-7e990af56058\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " Sep 10 16:40:33.577152 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.576964 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-home\") pod \"a43ee067-34a1-448d-ab0f-7e990af56058\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " Sep 10 16:40:33.577152 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.576991 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-tmp-dir\") pod \"a43ee067-34a1-448d-ab0f-7e990af56058\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " Sep 10 16:40:33.577152 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.577018 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43ee067-34a1-448d-ab0f-7e990af56058-tls-certs\") pod \"a43ee067-34a1-448d-ab0f-7e990af56058\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " Sep 10 16:40:33.577152 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.577069 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgpcr\" (UniqueName: \"kubernetes.io/projected/a43ee067-34a1-448d-ab0f-7e990af56058-kube-api-access-fgpcr\") pod \"a43ee067-34a1-448d-ab0f-7e990af56058\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " Sep 10 16:40:33.577152 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.577098 2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-dshm\") pod \"a43ee067-34a1-448d-ab0f-7e990af56058\" (UID: \"a43ee067-34a1-448d-ab0f-7e990af56058\") " Sep 10 16:40:33.577454 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.577289 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-model-cache" (OuterVolumeSpecName: "model-cache") pod "a43ee067-34a1-448d-ab0f-7e990af56058" (UID: "a43ee067-34a1-448d-ab0f-7e990af56058"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:40:33.577454 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.577371 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-home" (OuterVolumeSpecName: "home") pod "a43ee067-34a1-448d-ab0f-7e990af56058" (UID: "a43ee067-34a1-448d-ab0f-7e990af56058"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:40:33.577454 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.577411 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a43ee067-34a1-448d-ab0f-7e990af56058" (UID: "a43ee067-34a1-448d-ab0f-7e990af56058"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:40:33.577454 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.577427 2541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-home\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:40:33.577454 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.577439 2541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-model-cache\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:40:33.579691 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.579661 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-dshm" (OuterVolumeSpecName: "dshm") pod "a43ee067-34a1-448d-ab0f-7e990af56058" (UID: "a43ee067-34a1-448d-ab0f-7e990af56058"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:40:33.579691 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.579683 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a43ee067-34a1-448d-ab0f-7e990af56058-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a43ee067-34a1-448d-ab0f-7e990af56058" (UID: "a43ee067-34a1-448d-ab0f-7e990af56058"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:40:33.579861 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.579687 2541 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43ee067-34a1-448d-ab0f-7e990af56058-kube-api-access-fgpcr" (OuterVolumeSpecName: "kube-api-access-fgpcr") pod "a43ee067-34a1-448d-ab0f-7e990af56058" (UID: "a43ee067-34a1-448d-ab0f-7e990af56058"). InnerVolumeSpecName "kube-api-access-fgpcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:40:33.678164 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.678119 2541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43ee067-34a1-448d-ab0f-7e990af56058-tls-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:40:33.678164 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.678155 2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgpcr\" (UniqueName: \"kubernetes.io/projected/a43ee067-34a1-448d-ab0f-7e990af56058-kube-api-access-fgpcr\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:40:33.678164 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.678164 2541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-dshm\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:40:33.678164 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.678172 2541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43ee067-34a1-448d-ab0f-7e990af56058-tmp-dir\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\"" Sep 10 16:40:33.809552 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.809506 2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp"] Sep 10 16:40:33.819092 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:33.819059 2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-1d0deeff-kserve-84f84bd555-l45bp"] Sep 10 16:40:34.981347 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:34.981287 2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a43ee067-34a1-448d-ab0f-7e990af56058" path="/var/lib/kubelet/pods/a43ee067-34a1-448d-ab0f-7e990af56058/volumes" Sep 10 16:40:34.981798 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:34.981596 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:40:41.591813 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:41.591778 2541 scope.go:117] "RemoveContainer" containerID="d10b72980955c18d6ae439f2e9617c31c9992c3c829a6ac307da87c403168206" Sep 10 16:40:42.421800 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:42.421748 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:40:49.576827 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:49.576791 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:40:58.170363 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:40:58.170322 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:41:05.355898 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:05.355857 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gzlkl"] Sep 10 16:41:05.681800 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:05.681764 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:14.640661 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:14.640631 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-vncld_fde2043f-9788-47c7-a46e-3ef85064a65c/manager/0.log" Sep 10 16:41:15.166078 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:15.166004 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-jd8qq_65e8d3c8-3d65-41c9-83f2-3e881d1639a9/manager/0.log" Sep 10 16:41:15.287162 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:15.287135 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67779d4d4c-6jlgh_235f85ce-8fc5-44fd-9110-0fa535c2f76e/copy-manifests/0.log" Sep 10 16:41:15.298069 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:15.298040 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67779d4d4c-6jlgh_235f85ce-8fc5-44fd-9110-0fa535c2f76e/manager/0.log" Sep 10 16:41:17.711950 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:17.711875 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-98rc4_bc3a0463-7280-48ed-b8a7-38f5cd64b2d9/manager/0.log" Sep 10 16:41:17.835433 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:17.835402 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-gzlkl_c143d4ff-7d79-41b8-8898-dea4091300e1/limitador/0.log" Sep 10 16:41:17.954528 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:17.954491 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-ftxxj_a519db3c-1725-4d67-82fc-f88e519425aa/manager/0.log" Sep 10 16:41:18.449762 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:18.449722 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6bbs2_e1cf8d46-0b95-46ef-97d0-85545423d6e1/discovery/0.log" Sep 10 16:41:18.559395 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:18.559355 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74797cb87b-brnlx_6238467c-baff-455b-a7fc-aa900e8f9fc2/kube-auth-proxy/0.log" Sep 10 16:41:19.029959 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:19.029930 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-85d2w_c9c7e7f6-7f76-493c-aa7a-f69a70592836/payload-processing/0.log" Sep 10 16:41:19.158875 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:19.158849 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f48d5849b-h27lv_fdd6746a-7957-4de4-8241-4c1d740f7814/router/0.log" Sep 10 16:41:19.646744 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:19.646710 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-cd2ll_2b159143-5244-4115-80c0-79b2e9828784/main/0.log" Sep 10 16:41:20.118245 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:20.118145 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-wqxcl_c5b90458-3710-4957-8785-0300678d368a/main/0.log" Sep 10 16:41:25.554824 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:25.554794 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:26.786895 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:26.786861 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:29.185731 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:29.185686 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cjj7d_e3074957-6ddd-4ded-9ddf-7c2fe68929f0/global-pull-secret-syncer/0.log" Sep 10 16:41:29.273958 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:29.273918 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-48lj8_40f27517-24fa-41df-8a4d-1eefd387cf5d/konnectivity-agent/0.log" Sep 10 16:41:29.464917 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:29.464845 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-172.ec2.internal_93402cb6c1e59df178a12145f97bf120/haproxy/0.log" Sep 10 16:41:34.814261 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:34.814231 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-98rc4_bc3a0463-7280-48ed-b8a7-38f5cd64b2d9/manager/0.log" Sep 10 16:41:34.845060 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:34.845034 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-gzlkl_c143d4ff-7d79-41b8-8898-dea4091300e1/limitador/0.log" Sep 10 16:41:34.920139 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:34.920108 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-ftxxj_a519db3c-1725-4d67-82fc-f88e519425aa/manager/0.log" Sep 10 16:41:36.482077 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:36.482048 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f035c949-761e-4516-9b3e-1d48e505b056/alertmanager/0.log" Sep 10 16:41:36.555164 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:36.555089 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f035c949-761e-4516-9b3e-1d48e505b056/config-reloader/0.log" Sep 10 16:41:36.603847 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:36.603822 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f035c949-761e-4516-9b3e-1d48e505b056/kube-rbac-proxy-web/0.log" Sep 10 16:41:36.645769 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:36.645735 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f035c949-761e-4516-9b3e-1d48e505b056/kube-rbac-proxy/0.log" Sep 10 16:41:36.678736 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:36.678713 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f035c949-761e-4516-9b3e-1d48e505b056/kube-rbac-proxy-metric/0.log" Sep 10 16:41:36.718830 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:36.718802 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f035c949-761e-4516-9b3e-1d48e505b056/prom-label-proxy/0.log" Sep 10 16:41:36.755239 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:36.755196 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f035c949-761e-4516-9b3e-1d48e505b056/init-config-reloader/0.log" Sep 10 16:41:36.807937 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:36.807867 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-4nbnm_befb7be9-0852-4486-a536-219bb03d51f6/cluster-monitoring-operator/0.log" Sep 10 16:41:37.323478 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:37.323450 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vblpz_0ab59368-68c9-4b6b-a61f-6cf978e2fb30/node-exporter/0.log" Sep 10 16:41:37.366463 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:37.366428 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vblpz_0ab59368-68c9-4b6b-a61f-6cf978e2fb30/kube-rbac-proxy/0.log" Sep 10 16:41:37.398283 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:37.398259 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vblpz_0ab59368-68c9-4b6b-a61f-6cf978e2fb30/init-textfile/0.log" Sep 10 16:41:37.836286 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:37.836252 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-t2txz_8f659978-e772-408c-9ecf-59e161eab712/prometheus-operator/0.log" Sep 10 16:41:37.878840 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:37.878812 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-t2txz_8f659978-e772-408c-9ecf-59e161eab712/kube-rbac-proxy/0.log" Sep 10 16:41:38.741731 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.741697 2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt"] Sep 10 16:41:38.742103 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.742088 2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a43ee067-34a1-448d-ab0f-7e990af56058" containerName="main" Sep 10 16:41:38.742153 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.742105 2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="a43ee067-34a1-448d-ab0f-7e990af56058" containerName="main" Sep 10 16:41:38.742217 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.742205 2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="a43ee067-34a1-448d-ab0f-7e990af56058" containerName="main" Sep 10 16:41:38.745470 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.745448 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt" Sep 10 16:41:38.748890 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.748870 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dg8f5\"/\"kube-root-ca.crt\"" Sep 10 16:41:38.749255 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.749239 2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dg8f5\"/\"default-dockercfg-rxdwj\"" Sep 10 16:41:38.750332 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.750289 2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dg8f5\"/\"openshift-service-ca.crt\"" Sep 10 16:41:38.775794 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.775681 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt"] Sep 10 16:41:38.872774 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.872737 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ee6f3b9-f42a-431b-99a6-9c1c412e0359-lib-modules\") pod \"perf-node-gather-daemonset-78pvt\" (UID: \"6ee6f3b9-f42a-431b-99a6-9c1c412e0359\") " pod="openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt" Sep 10 16:41:38.872774 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.872776 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6ee6f3b9-f42a-431b-99a6-9c1c412e0359-proc\") pod \"perf-node-gather-daemonset-78pvt\" (UID: \"6ee6f3b9-f42a-431b-99a6-9c1c412e0359\") " pod="openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt" Sep 10 16:41:38.873223 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.872793 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ee6f3b9-f42a-431b-99a6-9c1c412e0359-sys\") pod \"perf-node-gather-daemonset-78pvt\" (UID: \"6ee6f3b9-f42a-431b-99a6-9c1c412e0359\") " pod="openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt" Sep 10 16:41:38.873223 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.872879 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6ee6f3b9-f42a-431b-99a6-9c1c412e0359-podres\") pod \"perf-node-gather-daemonset-78pvt\" (UID: \"6ee6f3b9-f42a-431b-99a6-9c1c412e0359\") " pod="openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt" Sep 10 16:41:38.873223 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.872903 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4flhs\" (UniqueName: \"kubernetes.io/projected/6ee6f3b9-f42a-431b-99a6-9c1c412e0359-kube-api-access-4flhs\") pod \"perf-node-gather-daemonset-78pvt\" (UID: \"6ee6f3b9-f42a-431b-99a6-9c1c412e0359\") " pod="openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt" Sep 10 16:41:38.974163 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.974133 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ee6f3b9-f42a-431b-99a6-9c1c412e0359-lib-modules\") pod \"perf-node-gather-daemonset-78pvt\" (UID: \"6ee6f3b9-f42a-431b-99a6-9c1c412e0359\") " pod="openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt" Sep 10 16:41:38.974163 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.974170 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6ee6f3b9-f42a-431b-99a6-9c1c412e0359-proc\") pod \"perf-node-gather-daemonset-78pvt\" (UID: \"6ee6f3b9-f42a-431b-99a6-9c1c412e0359\") " pod="openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt" Sep 10 16:41:38.974467 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.974185 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ee6f3b9-f42a-431b-99a6-9c1c412e0359-sys\") pod \"perf-node-gather-daemonset-78pvt\" (UID: \"6ee6f3b9-f42a-431b-99a6-9c1c412e0359\") " pod="openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt" Sep 10 16:41:38.974467 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.974207 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6ee6f3b9-f42a-431b-99a6-9c1c412e0359-podres\") pod \"perf-node-gather-daemonset-78pvt\" (UID: \"6ee6f3b9-f42a-431b-99a6-9c1c412e0359\") " pod="openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt" Sep 10 16:41:38.974467 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.974232 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4flhs\" (UniqueName: \"kubernetes.io/projected/6ee6f3b9-f42a-431b-99a6-9c1c412e0359-kube-api-access-4flhs\") pod \"perf-node-gather-daemonset-78pvt\" (UID: \"6ee6f3b9-f42a-431b-99a6-9c1c412e0359\") " pod="openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt" Sep 10 16:41:38.974467 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.974292 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ee6f3b9-f42a-431b-99a6-9c1c412e0359-sys\") pod \"perf-node-gather-daemonset-78pvt\" (UID: \"6ee6f3b9-f42a-431b-99a6-9c1c412e0359\") " pod="openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt" Sep 10 16:41:38.974467 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.974391 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6ee6f3b9-f42a-431b-99a6-9c1c412e0359-proc\") pod \"perf-node-gather-daemonset-78pvt\" (UID: \"6ee6f3b9-f42a-431b-99a6-9c1c412e0359\") " pod="openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt" Sep 10 16:41:38.974467 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.974394 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ee6f3b9-f42a-431b-99a6-9c1c412e0359-lib-modules\") pod \"perf-node-gather-daemonset-78pvt\" (UID: \"6ee6f3b9-f42a-431b-99a6-9c1c412e0359\") " pod="openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt" Sep 10 16:41:38.974467 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.974428 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6ee6f3b9-f42a-431b-99a6-9c1c412e0359-podres\") pod \"perf-node-gather-daemonset-78pvt\" (UID: \"6ee6f3b9-f42a-431b-99a6-9c1c412e0359\") " pod="openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt" Sep 10 16:41:38.990926 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:38.990904 2541 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4flhs\" (UniqueName: \"kubernetes.io/projected/6ee6f3b9-f42a-431b-99a6-9c1c412e0359-kube-api-access-4flhs\") pod \"perf-node-gather-daemonset-78pvt\" (UID: \"6ee6f3b9-f42a-431b-99a6-9c1c412e0359\") " pod="openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt" Sep 10 16:41:39.055653 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:39.055550 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt" Sep 10 16:41:39.343269 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:39.343193 2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt"] Sep 10 16:41:39.346722 ip-10-0-137-172 kubenswrapper[2541]: W0910 16:41:39.346691 2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6ee6f3b9_f42a_431b_99a6_9c1c412e0359.slice/crio-e062a314f4304acce6f725b77962b70aa40021b520d6c7eb9f9df13e415cca6a WatchSource:0}: Error finding container e062a314f4304acce6f725b77962b70aa40021b520d6c7eb9f9df13e415cca6a: Status 404 returned error can't find the container with id e062a314f4304acce6f725b77962b70aa40021b520d6c7eb9f9df13e415cca6a Sep 10 16:41:39.542258 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:39.542223 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-c4p62_30ac85d4-8250-4700-9d2b-c19e029b3ddc/networking-console-plugin/0.log" Sep 10 16:41:39.847838 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:39.847801 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt" event={"ID":"6ee6f3b9-f42a-431b-99a6-9c1c412e0359","Type":"ContainerStarted","Data":"a75a9b99a534b03a9b8dfe142d9852f208efcd9081464602bb861480871f2905"} Sep 10 16:41:39.848041 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:39.847860 2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt" event={"ID":"6ee6f3b9-f42a-431b-99a6-9c1c412e0359","Type":"ContainerStarted","Data":"e062a314f4304acce6f725b77962b70aa40021b520d6c7eb9f9df13e415cca6a"} Sep 10 16:41:39.848041 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:39.847888 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt" Sep 10 16:41:39.874486 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:39.874440 2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt" podStartSLOduration=1.874424264 podStartE2EDuration="1.874424264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:41:39.872494337 +0000 UTC m=+2399.395739533" watchObservedRunningTime="2026-09-10 16:41:39.874424264 +0000 UTC m=+2399.397669460" Sep 10 16:41:40.330472 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:40.330440 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/1.log" Sep 10 16:41:40.334293 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:40.334269 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/2.log" Sep 10 16:41:40.954202 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:40.954176 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-vmph9_03fe5d54-88f0-49bf-ba0b-96e56def25dc/download-server/0.log" Sep 10 16:41:41.225879 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:41.225797 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/1.log" Sep 10 16:41:41.231400 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:41.231364 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bt6g6_70b8f4ef-c89f-48fe-b006-de3f7f6d6d3b/console-operator/1.log" Sep 10 16:41:41.231943 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:41.231910 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4c6j_2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd/ovn-acl-logging/0.log" Sep 10 16:41:41.237558 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:41.237538 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4c6j_2d7c8cbd-bcf1-4ef4-8091-8d9333f251fd/ovn-acl-logging/0.log" Sep 10 16:41:41.648812 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:41.648784 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-scdqf_ae665a3c-336f-4057-8e8e-ca9572f1fd58/volume-data-source-validator/0.log" Sep 10 16:41:42.949914 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:42.949882 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6r7v6_0c3d3154-d3b2-4886-9bb6-0036bc9bdd35/dns/0.log" Sep 10 16:41:42.983132 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:42.983107 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6r7v6_0c3d3154-d3b2-4886-9bb6-0036bc9bdd35/kube-rbac-proxy/0.log" Sep 10 16:41:43.177772 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:43.177739 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xbqb7_ac6acd6c-a24c-47cc-a461-778f88278b3c/dns-node-resolver/0.log" Sep 10 16:41:43.949828 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:43.949795 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rxmz9_7e5689cb-d3f5-4d8c-b91c-a0e40aeeeef9/node-ca/0.log" Sep 10 16:41:45.333953 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:45.333922 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6bbs2_e1cf8d46-0b95-46ef-97d0-85545423d6e1/discovery/0.log" Sep 10 16:41:45.367796 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:45.367765 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74797cb87b-brnlx_6238467c-baff-455b-a7fc-aa900e8f9fc2/kube-auth-proxy/0.log" Sep 10 16:41:45.546016 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:45.545988 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-85d2w_c9c7e7f6-7f76-493c-aa7a-f69a70592836/payload-processing/0.log" Sep 10 16:41:45.589637 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:45.589559 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f48d5849b-h27lv_fdd6746a-7957-4de4-8241-4c1d740f7814/router/0.log" Sep 10 16:41:45.864005 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:45.863921 2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dg8f5/perf-node-gather-daemonset-78pvt" Sep 10 16:41:46.581166 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:46.581126 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pk2xn_b0e776dd-a8d0-4527-b511-729ca90900bb/serve-healthcheck-canary/0.log" Sep 10 16:41:47.327025 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:47.326999 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-89xqs_3a5e2ec6-0e23-4d13-a153-25d193930e01/insights-operator/0.log" Sep 10 16:41:47.327509 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:47.327474 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-89xqs_3a5e2ec6-0e23-4d13-a153-25d193930e01/insights-operator/1.log" Sep 10 16:41:47.677720 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:47.677696 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ztp4g_946ce162-a05d-4e23-9c7c-5be6a51e0595/kube-rbac-proxy/0.log" Sep 10 16:41:47.719551 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:47.719524 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ztp4g_946ce162-a05d-4e23-9c7c-5be6a51e0595/exporter/0.log" Sep 10 16:41:47.765330 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:47.765289 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ztp4g_946ce162-a05d-4e23-9c7c-5be6a51e0595/extractor/0.log" Sep 10 16:41:49.619619 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:49.619532 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6bbs2_e1cf8d46-0b95-46ef-97d0-85545423d6e1/discovery/0.log" Sep 10 16:41:49.631231 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:49.631205 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74797cb87b-brnlx_6238467c-baff-455b-a7fc-aa900e8f9fc2/kube-auth-proxy/0.log" Sep 10 16:41:49.748584 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:49.748553 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-85d2w_c9c7e7f6-7f76-493c-aa7a-f69a70592836/payload-processing/0.log" Sep 10 16:41:49.765097 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:49.765067 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f48d5849b-h27lv_fdd6746a-7957-4de4-8241-4c1d740f7814/router/0.log" Sep 10 16:41:50.466250 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:50.466224 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-vncld_fde2043f-9788-47c7-a46e-3ef85064a65c/manager/0.log" Sep 10 16:41:50.860241 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:50.860215 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-jd8qq_65e8d3c8-3d65-41c9-83f2-3e881d1639a9/manager/0.log" Sep 10 16:41:50.894067 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:50.894035 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67779d4d4c-6jlgh_235f85ce-8fc5-44fd-9110-0fa535c2f76e/manager/0.log" Sep 10 16:41:50.936323 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:50.936274 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67779d4d4c-6jlgh_235f85ce-8fc5-44fd-9110-0fa535c2f76e/copy-manifests/0.log" Sep 10 16:41:51.637947 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:51.637916 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6bbs2_e1cf8d46-0b95-46ef-97d0-85545423d6e1/discovery/0.log" Sep 10 16:41:51.656947 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:51.656923 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74797cb87b-brnlx_6238467c-baff-455b-a7fc-aa900e8f9fc2/kube-auth-proxy/0.log" Sep 10 16:41:51.801864 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:51.801819 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-85d2w_c9c7e7f6-7f76-493c-aa7a-f69a70592836/payload-processing/0.log" Sep 10 16:41:51.823089 ip-10-0-137-172 kubenswrapper[2541]: I0910 16:41:51.823061 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f48d5849b-h27lv_fdd6746a-7957-4de4-8241-4c1d740f7814/router/0.log"