Sep 16 13:30:51.790872 ip-10-0-132-9 systemd[1]: Starting Kubernetes Kubelet... Sep 16 13:30:52.165388 ip-10-0-132-9 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 13:30:52.165388 ip-10-0-132-9 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 13:30:52.165388 ip-10-0-132-9 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 13:30:52.165388 ip-10-0-132-9 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 13:30:52.165388 ip-10-0-132-9 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 13:30:52.166077 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.165385 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 13:30:52.169333 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169317 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:30:52.169374 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169334 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:30:52.169374 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169339 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:30:52.169374 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169343 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:30:52.169374 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169347 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:30:52.169374 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169350 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:30:52.169374 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169353 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:30:52.169374 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169356 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:30:52.169374 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169360 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:30:52.169374 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169364 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:30:52.169374 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169367 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:30:52.169374 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169370 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:30:52.169374 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169372 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:30:52.169374 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169375 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:30:52.169374 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169378 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:30:52.169374 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169381 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:30:52.169767 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169391 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:30:52.169767 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169394 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:30:52.169767 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169397 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:30:52.169767 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169399 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:30:52.169767 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169401 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:30:52.169767 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169404 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:30:52.169767 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169407 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:30:52.169767 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169409 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:30:52.169767 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169412 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:30:52.169767 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169414 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:30:52.169767 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169416 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:30:52.169767 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169419 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:30:52.169767 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169422 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:30:52.169767 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169424 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:30:52.169767 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169427 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:30:52.169767 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169429 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:30:52.169767 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169432 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:30:52.169767 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169434 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:30:52.169767 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169437 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:30:52.169767 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169440 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:30:52.170260 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169442 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:30:52.170260 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169445 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:30:52.170260 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169447 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:30:52.170260 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169450 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:30:52.170260 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169452 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:30:52.170260 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169455 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:30:52.170260 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169457 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:30:52.170260 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169461 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:30:52.170260 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169463 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:30:52.170260 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169466 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:30:52.170260 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169468 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:30:52.170260 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169471 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:30:52.170260 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169474 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:30:52.170260 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169483 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:30:52.170260 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169488 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:30:52.170260 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169491 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:30:52.170260 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169494 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:30:52.170260 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169496 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:30:52.170260 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169499 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:30:52.170260 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169502 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:30:52.170803 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169504 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:30:52.170803 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169507 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:30:52.170803 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169509 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:30:52.170803 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169512 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:30:52.170803 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169514 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:30:52.170803 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169517 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:30:52.170803 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169520 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:30:52.170803 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169522 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:30:52.170803 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169526 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:30:52.170803 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169529 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:30:52.170803 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169531 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:30:52.170803 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169534 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:30:52.170803 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169538 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:30:52.170803 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169541 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:30:52.170803 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169544 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:30:52.170803 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169547 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:30:52.170803 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169549 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:30:52.170803 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169552 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:30:52.170803 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169555 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:30:52.170803 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169557 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:30:52.171291 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169560 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:30:52.171291 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169562 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:30:52.171291 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169565 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:30:52.171291 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169568 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:30:52.171291 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169570 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:30:52.171291 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169575 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:30:52.171291 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169577 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:30:52.171291 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169580 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:30:52.171291 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169582 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:30:52.171291 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169585 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:30:52.171291 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169587 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:30:52.171291 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169590 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:30:52.171291 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169592 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:30:52.171291 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169595 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:30:52.171291 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169597 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:30:52.171291 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169599 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:30:52.171291 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169617 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:30:52.171291 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169621 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:30:52.171291 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169626 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:30:52.171291 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169630 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:30:52.171821 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169633 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:30:52.171821 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169635 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:30:52.171821 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.169638 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:30:52.171821 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170314 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:30:52.171821 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170321 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:30:52.171821 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170324 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:30:52.171821 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170326 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:30:52.171821 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170329 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:30:52.171821 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170332 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:30:52.171821 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170335 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:30:52.171821 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170337 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:30:52.171821 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170340 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:30:52.171821 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170343 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:30:52.171821 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170347 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:30:52.171821 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170351 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:30:52.171821 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170354 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:30:52.171821 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170363 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:30:52.171821 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170368 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:30:52.171821 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170371 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:30:52.172325 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170374 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:30:52.172325 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170377 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:30:52.172325 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170379 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:30:52.172325 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170382 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:30:52.172325 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170385 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:30:52.172325 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170387 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:30:52.172325 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170390 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:30:52.172325 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170392 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:30:52.172325 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170395 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:30:52.172325 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170397 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:30:52.172325 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170400 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:30:52.172325 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170402 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:30:52.172325 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170405 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:30:52.172325 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170407 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:30:52.172325 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170409 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:30:52.172325 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170412 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:30:52.172325 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170414 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:30:52.172325 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170417 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:30:52.172325 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170419 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:30:52.172325 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170422 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:30:52.172837 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170425 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:30:52.172837 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170427 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:30:52.172837 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170429 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:30:52.172837 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170432 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:30:52.172837 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170434 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:30:52.172837 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170437 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:30:52.172837 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170439 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:30:52.172837 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170441 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:30:52.172837 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170444 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:30:52.172837 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170446 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:30:52.172837 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170450 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:30:52.172837 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170453 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:30:52.172837 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170456 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:30:52.172837 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170458 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:30:52.172837 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170462 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:30:52.172837 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170465 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:30:52.172837 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170467 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:30:52.172837 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170469 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:30:52.172837 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170472 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:30:52.172837 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170474 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:30:52.173322 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170476 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:30:52.173322 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170479 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:30:52.173322 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170481 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:30:52.173322 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170484 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:30:52.173322 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170488 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:30:52.173322 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170491 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:30:52.173322 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170494 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:30:52.173322 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170496 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:30:52.173322 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170499 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:30:52.173322 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170504 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:30:52.173322 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170507 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:30:52.173322 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170510 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:30:52.173322 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170513 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:30:52.173322 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170516 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:30:52.173322 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170518 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:30:52.173322 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170521 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:30:52.173322 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170524 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:30:52.173322 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170527 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:30:52.173322 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170529 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:30:52.173815 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170532 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:30:52.173815 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170534 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:30:52.173815 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170537 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:30:52.173815 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170541 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:30:52.173815 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170545 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:30:52.173815 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170547 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:30:52.173815 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170550 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:30:52.173815 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170552 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:30:52.173815 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170555 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:30:52.173815 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170557 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:30:52.173815 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170560 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:30:52.173815 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170562 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:30:52.173815 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170565 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:30:52.173815 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170567 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:30:52.173815 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170570 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:30:52.173815 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170572 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:30:52.173815 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170574 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:30:52.173815 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170577 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:30:52.173815 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170579 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:30:52.173815 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170582 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:30:52.174326 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170584 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:30:52.174326 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170586 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:30:52.174326 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170589 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:30:52.174326 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170591 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:30:52.174326 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170964 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:30:52.174326 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170969 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:30:52.174326 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170972 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:30:52.174326 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170974 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:30:52.174326 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170977 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:30:52.174326 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170980 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:30:52.174326 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170982 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:30:52.174326 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170985 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:30:52.174326 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170987 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:30:52.174326 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170990 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:30:52.174326 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170992 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:30:52.174326 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170996 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:30:52.174326 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.170999 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:30:52.174326 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171002 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:30:52.174326 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171004 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:30:52.174326 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171007 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:30:52.174843 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171009 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:30:52.174843 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171012 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:30:52.174843 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171014 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:30:52.174843 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171016 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:30:52.174843 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171019 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:30:52.174843 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171021 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:30:52.174843 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171024 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:30:52.174843 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171026 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:30:52.174843 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171029 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:30:52.174843 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171031 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:30:52.174843 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171034 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:30:52.174843 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171036 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:30:52.174843 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171039 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:30:52.174843 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171041 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:30:52.174843 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171044 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:30:52.174843 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171046 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:30:52.174843 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171048 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:30:52.174843 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171051 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:30:52.174843 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171053 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:30:52.174843 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171056 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:30:52.175345 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171059 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:30:52.175345 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171061 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:30:52.175345 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171065 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:30:52.175345 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171068 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:30:52.175345 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171071 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:30:52.175345 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171074 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:30:52.175345 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171076 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:30:52.175345 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171079 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:30:52.175345 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171082 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:30:52.175345 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171085 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:30:52.175345 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171089 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:30:52.175345 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171092 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:30:52.175345 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171095 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:30:52.175345 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171098 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:30:52.175345 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171101 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:30:52.175345 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171103 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:30:52.175345 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171106 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:30:52.175345 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171108 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:30:52.175345 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171111 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:30:52.175839 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171113 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:30:52.175839 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171115 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:30:52.175839 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171118 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:30:52.175839 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171121 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:30:52.175839 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171123 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:30:52.175839 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171126 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:30:52.175839 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171128 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:30:52.175839 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171131 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:30:52.175839 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171133 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:30:52.175839 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171136 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:30:52.175839 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171138 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:30:52.175839 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171140 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:30:52.175839 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171143 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:30:52.175839 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171145 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:30:52.175839 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171148 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:30:52.175839 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171150 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:30:52.175839 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171152 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:30:52.175839 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171155 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:30:52.175839 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171158 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:30:52.175839 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171160 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:30:52.176320 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171165 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:30:52.176320 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171168 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:30:52.176320 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171171 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:30:52.176320 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171174 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:30:52.176320 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171177 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:30:52.176320 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171179 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:30:52.176320 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171181 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:30:52.176320 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171184 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:30:52.176320 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171186 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:30:52.176320 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171189 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:30:52.176320 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171191 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:30:52.176320 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171194 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:30:52.176320 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171196 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:30:52.176320 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171199 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:30:52.176320 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171202 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:30:52.176320 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171204 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:30:52.176320 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171206 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:30:52.176320 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171209 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:30:52.176320 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171211 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171214 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171216 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171219 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171221 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171223 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171295 2580 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171330 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171336 2580 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171340 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171344 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171349 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171354 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171359 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171363 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171366 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171369 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171372 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171376 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171379 2580 flags.go:64] FLAG: --cgroup-root="" Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171382 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171385 2580 flags.go:64] FLAG: --client-ca-file="" Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171388 2580 flags.go:64] FLAG: --cloud-provider="external" Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171390 2580 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 13:30:52.176786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171395 2580 flags.go:64] FLAG: --cluster-domain="" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171397 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171400 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171404 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171407 2580 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171410 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171413 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171416 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171419 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171422 2580 flags.go:64] FLAG: --contention-profiling="false" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171425 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171428 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171431 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171434 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171438 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171440 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171443 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171446 2580 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171449 2580 flags.go:64] FLAG: --enable-server="true" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171453 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171457 2580 flags.go:64] FLAG: --event-burst="100" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171460 2580 flags.go:64] FLAG: --event-qps="50" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171465 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171468 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 13:30:52.177470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171471 2580 flags.go:64] FLAG: --eviction-hard="" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171475 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171478 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171481 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171484 2580 flags.go:64] FLAG: --eviction-soft="" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171487 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171490 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171492 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171495 2580 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171498 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171500 2580 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171503 2580 flags.go:64] FLAG: --feature-gates="" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171507 2580 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171509 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171512 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171515 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171518 2580 flags.go:64] FLAG: --healthz-port="10248" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171521 2580 flags.go:64] FLAG: --help="false" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171523 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-132-9.ec2.internal" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171526 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171529 2580 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171532 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171535 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 13:30:52.178127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171538 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171541 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171544 2580 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171546 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171550 2580 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171553 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171556 2580 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171560 2580 flags.go:64] FLAG: --kube-reserved="" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171563 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171566 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171568 2580 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171571 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171574 2580 flags.go:64] FLAG: --lock-file="" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171577 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171579 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171582 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171587 2580 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171590 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171593 2580 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171596 2580 flags.go:64] FLAG: --logging-format="text" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171599 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171617 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171620 2580 flags.go:64] FLAG: --manifest-url="" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171623 2580 flags.go:64] FLAG: --manifest-url-header="" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171627 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 13:30:52.178831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171630 2580 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171634 2580 flags.go:64] FLAG: --max-pods="110" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171637 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171639 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171642 2580 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171645 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171648 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171650 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171653 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171660 2580 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171663 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171667 2580 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171670 2580 flags.go:64] FLAG: --pod-cidr="" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171673 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171678 2580 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171682 2580 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171685 2580 flags.go:64] FLAG: --pods-per-core="0" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171689 2580 flags.go:64] FLAG: --port="10250" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171691 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171694 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b079961e918581c4" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171697 2580 flags.go:64] FLAG: --qos-reserved="" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171701 2580 flags.go:64] FLAG: --read-only-port="10255" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171704 2580 flags.go:64] FLAG: --register-node="true" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171706 2580 flags.go:64] FLAG: --register-with-taints="" Sep 16 13:30:52.179439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171710 2580 flags.go:64] FLAG: --registry-burst="10" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171712 2580 flags.go:64] FLAG: --registry-qps="5" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171715 2580 flags.go:64] FLAG: --reserved-cpus="" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171718 2580 flags.go:64] FLAG: --reserved-memory="" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171721 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171724 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171727 2580 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171730 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171732 2580 flags.go:64] FLAG: --runonce="false" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171735 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171738 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171741 2580 flags.go:64] FLAG: --seccomp-default="false" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171744 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171746 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171749 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171752 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171755 2580 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171757 2580 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171760 2580 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171764 2580 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171766 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171769 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171772 2580 flags.go:64] FLAG: --system-cgroups="" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171777 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171782 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 13:30:52.180037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171784 2580 flags.go:64] FLAG: --tls-cert-file="" Sep 16 13:30:52.180671 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171787 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 13:30:52.180671 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171791 2580 flags.go:64] FLAG: --tls-min-version="" Sep 16 13:30:52.180671 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171793 2580 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 13:30:52.180671 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171796 2580 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 13:30:52.180671 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171799 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 13:30:52.180671 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171802 2580 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 13:30:52.180671 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171805 2580 flags.go:64] FLAG: --v="2" Sep 16 13:30:52.180671 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171809 2580 flags.go:64] FLAG: --version="false" Sep 16 13:30:52.180671 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171813 2580 flags.go:64] FLAG: --vmodule="" Sep 16 13:30:52.180671 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171816 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 13:30:52.180671 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.171819 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 13:30:52.180671 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171929 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:30:52.180671 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171933 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:30:52.180671 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171936 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:30:52.180671 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171939 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:30:52.180671 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171942 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:30:52.180671 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171945 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:30:52.180671 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171948 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:30:52.180671 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171952 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:30:52.180671 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171954 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:30:52.180671 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171957 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:30:52.180671 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171960 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:30:52.181193 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171963 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:30:52.181193 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171965 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:30:52.181193 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171968 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:30:52.181193 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171972 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:30:52.181193 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171975 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:30:52.181193 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171977 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:30:52.181193 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171980 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:30:52.181193 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171985 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:30:52.181193 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171988 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:30:52.181193 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171991 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:30:52.181193 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171994 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:30:52.181193 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171997 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:30:52.181193 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.171999 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:30:52.181193 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172002 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:30:52.181193 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172004 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:30:52.181193 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172007 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:30:52.181193 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172010 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:30:52.181193 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172013 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:30:52.181193 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172015 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:30:52.181193 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172018 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:30:52.181755 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172021 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:30:52.181755 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172023 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:30:52.181755 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172025 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:30:52.181755 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172028 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:30:52.181755 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172030 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:30:52.181755 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172033 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:30:52.181755 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172035 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:30:52.181755 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172038 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:30:52.181755 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172040 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:30:52.181755 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172042 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:30:52.181755 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172046 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:30:52.181755 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172050 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:30:52.181755 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172053 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:30:52.181755 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172056 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:30:52.181755 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172058 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:30:52.181755 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172062 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:30:52.181755 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172065 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:30:52.181755 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172067 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:30:52.181755 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172070 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:30:52.182220 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172073 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:30:52.182220 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172076 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:30:52.182220 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172078 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:30:52.182220 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172081 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:30:52.182220 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172083 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:30:52.182220 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172086 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:30:52.182220 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172088 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:30:52.182220 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172091 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:30:52.182220 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172094 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:30:52.182220 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172098 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:30:52.182220 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172101 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:30:52.182220 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172104 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:30:52.182220 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172107 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:30:52.182220 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172109 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:30:52.182220 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172112 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:30:52.182220 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172114 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:30:52.182220 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172117 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:30:52.182220 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172119 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:30:52.182220 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172122 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:30:52.182220 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172124 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:30:52.182820 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172127 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:30:52.182820 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172129 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:30:52.182820 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172132 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:30:52.182820 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172134 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:30:52.182820 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172137 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:30:52.182820 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172139 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:30:52.182820 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172142 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:30:52.182820 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172144 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:30:52.182820 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172148 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:30:52.182820 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172152 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:30:52.182820 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172156 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:30:52.182820 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172158 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:30:52.182820 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172162 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:30:52.182820 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172164 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:30:52.182820 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172167 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:30:52.182820 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172169 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:30:52.182820 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172172 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:30:52.182820 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172175 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:30:52.182820 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172179 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:30:52.182820 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172183 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:30:52.183321 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172187 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:30:52.183321 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172190 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:30:52.183321 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172195 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:30:52.183321 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172198 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:30:52.183321 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172200 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:30:52.183321 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172203 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:30:52.183321 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172205 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:30:52.183321 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172208 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:30:52.183321 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.172210 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:30:52.183321 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.172955 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 13:30:52.183321 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.179843 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 13:30:52.183321 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.179858 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 13:30:52.183321 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.179872 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 13:30:52.183321 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.179880 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 13:30:52.183321 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.179944 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:30:52.183733 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.179951 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:30:52.183733 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.179955 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:30:52.183733 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.179960 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:30:52.183733 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.179964 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:30:52.183733 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.179967 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:30:52.183733 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.179970 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:30:52.183733 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.179972 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:30:52.183733 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.179975 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:30:52.183733 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.179978 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:30:52.183733 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.179981 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:30:52.183733 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.179983 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:30:52.183733 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.179986 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:30:52.183733 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.179988 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:30:52.183733 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.179991 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:30:52.183733 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.179993 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:30:52.183733 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.179996 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:30:52.183733 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.179999 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:30:52.183733 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180001 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:30:52.183733 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180003 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:30:52.183733 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180006 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:30:52.184208 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180009 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:30:52.184208 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180011 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:30:52.184208 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180014 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:30:52.184208 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180016 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:30:52.184208 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180019 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:30:52.184208 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180022 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:30:52.184208 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180024 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:30:52.184208 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180027 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:30:52.184208 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180038 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:30:52.184208 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180041 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:30:52.184208 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180044 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:30:52.184208 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180046 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:30:52.184208 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180049 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:30:52.184208 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180052 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:30:52.184208 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180054 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:30:52.184208 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180057 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:30:52.184208 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180059 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:30:52.184208 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180062 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:30:52.184208 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180066 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:30:52.184208 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180070 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:30:52.184719 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180073 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:30:52.184719 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180075 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:30:52.184719 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180078 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:30:52.184719 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180080 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:30:52.184719 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180083 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:30:52.184719 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180085 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:30:52.184719 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180088 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:30:52.184719 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180090 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:30:52.184719 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180093 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:30:52.184719 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180095 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:30:52.184719 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180098 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:30:52.184719 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180101 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:30:52.184719 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180104 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:30:52.184719 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180106 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:30:52.184719 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180109 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:30:52.184719 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180111 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:30:52.184719 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180114 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:30:52.184719 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180117 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:30:52.184719 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180119 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:30:52.184719 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180121 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:30:52.185250 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180125 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:30:52.185250 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180133 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:30:52.185250 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180136 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:30:52.185250 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180138 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:30:52.185250 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180140 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:30:52.185250 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180143 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:30:52.185250 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180145 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:30:52.185250 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180148 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:30:52.185250 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180150 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:30:52.185250 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180154 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:30:52.185250 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180158 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:30:52.185250 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180161 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:30:52.185250 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180164 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:30:52.185250 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180167 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:30:52.185250 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180169 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:30:52.185250 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180172 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:30:52.185250 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180175 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:30:52.185250 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180178 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:30:52.185250 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180181 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:30:52.185741 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180184 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:30:52.185741 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180186 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:30:52.185741 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180189 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:30:52.185741 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180192 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:30:52.185741 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180194 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:30:52.185741 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180197 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:30:52.185741 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180200 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:30:52.185741 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180203 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:30:52.185741 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180205 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:30:52.185741 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180208 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:30:52.185741 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180212 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:30:52.185741 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180215 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:30:52.185741 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180218 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:30:52.185741 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180222 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:30:52.185741 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180224 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:30:52.185741 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180233 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:30:52.185741 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180235 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:30:52.185741 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180238 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:30:52.185741 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180240 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:30:52.186198 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.180245 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 13:30:52.186198 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180383 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:30:52.186198 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180388 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:30:52.186198 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180391 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:30:52.186198 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180394 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:30:52.186198 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180397 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:30:52.186198 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180399 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:30:52.186198 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180402 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:30:52.186198 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180405 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:30:52.186198 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180407 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:30:52.186198 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180410 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:30:52.186198 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180413 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:30:52.186198 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180415 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:30:52.186198 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180418 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:30:52.186198 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180420 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:30:52.186569 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180423 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:30:52.186569 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180425 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:30:52.186569 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180428 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:30:52.186569 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180430 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:30:52.186569 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180434 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:30:52.186569 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180436 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:30:52.186569 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180439 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:30:52.186569 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180441 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:30:52.186569 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180444 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:30:52.186569 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180447 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:30:52.186569 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180449 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:30:52.186569 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180452 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:30:52.186569 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180454 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:30:52.186569 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180457 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:30:52.186569 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180464 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:30:52.186569 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180467 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:30:52.186569 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180470 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:30:52.186569 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180478 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:30:52.186569 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180480 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:30:52.186569 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180483 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:30:52.187110 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180486 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:30:52.187110 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180488 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:30:52.187110 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180491 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:30:52.187110 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180493 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:30:52.187110 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180496 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:30:52.187110 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180498 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:30:52.187110 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180502 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:30:52.187110 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180506 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:30:52.187110 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180509 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:30:52.187110 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180511 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:30:52.187110 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180514 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:30:52.187110 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180517 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:30:52.187110 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180519 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:30:52.187110 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180522 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:30:52.187110 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180524 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:30:52.187110 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180527 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:30:52.187110 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180530 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:30:52.187110 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180532 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:30:52.187110 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180534 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:30:52.187110 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180537 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:30:52.187588 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180540 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:30:52.187588 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180544 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:30:52.187588 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180547 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:30:52.187588 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180549 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:30:52.187588 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180552 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:30:52.187588 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180554 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:30:52.187588 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180556 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:30:52.187588 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180564 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:30:52.187588 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180567 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:30:52.187588 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180569 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:30:52.187588 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180572 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:30:52.187588 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180574 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:30:52.187588 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180577 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:30:52.187588 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180579 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:30:52.187588 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180581 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:30:52.187588 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180584 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:30:52.187588 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180586 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:30:52.187588 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180589 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:30:52.187588 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180592 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:30:52.188059 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180595 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:30:52.188059 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180598 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:30:52.188059 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180614 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:30:52.188059 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180617 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:30:52.188059 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180619 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:30:52.188059 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180622 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:30:52.188059 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180624 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:30:52.188059 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180627 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:30:52.188059 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180629 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:30:52.188059 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180632 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:30:52.188059 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180635 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:30:52.188059 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180637 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:30:52.188059 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180640 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:30:52.188059 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180642 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:30:52.188059 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180645 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:30:52.188059 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180647 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:30:52.188059 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180650 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:30:52.188059 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180652 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:30:52.188059 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180655 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:30:52.188059 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180657 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:30:52.188664 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180659 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:30:52.188664 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180667 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:30:52.188664 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180670 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:30:52.188664 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180672 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:30:52.188664 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180675 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:30:52.188664 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:52.180678 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:30:52.188664 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.180682 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 13:30:52.188664 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.180817 2580 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 13:30:52.188664 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.182771 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 13:30:52.188664 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.183919 2580 server.go:1019] "Starting client certificate rotation" Sep 16 13:30:52.188664 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.184035 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 13:30:52.188664 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.184788 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 13:30:52.188664 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.186593 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 13:30:52.193970 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.193946 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 13:30:52.203927 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.203909 2580 log.go:25] "Validated CRI v1 runtime API" Sep 16 13:30:52.206688 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.206675 2580 log.go:25] "Validated CRI v1 image API" Sep 16 13:30:52.207515 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.207500 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 13:30:52.210586 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.210563 2580 fs.go:135] Filesystem UUIDs: map[12ab57a0-1fd7-47dd-aa3e-ce365fd35d07:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 88751482-6a2a-4594-9d3d-f68f200ad037:/dev/nvme0n1p4] Sep 16 13:30:52.210586 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.210580 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 13:30:52.214479 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.214459 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 13:30:52.215915 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.215797 2580 manager.go:216] Machine: {Timestamp:2026-09-16 13:30:52.21450243 +0000 UTC m=+0.315315137 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100988 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec201776109f5b500795147eb5222760 SystemUUID:ec201776-109f-5b50-0795-147eb5222760 BootID:ef6fc20a-6a79-4d43-acee-42bc522b8a12 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e6:0d:df:25 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e6:0d:df:25 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:d3:fb:f2:59:3d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 13:30:52.215915 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.215912 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 13:30:52.216030 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.215986 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 13:30:52.218896 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.218869 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 13:30:52.219028 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.218899 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-9.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 13:30:52.219480 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.219471 2580 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 13:30:52.219514 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.219483 2580 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 13:30:52.219514 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.219491 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 13:30:52.220008 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.219998 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 13:30:52.220716 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.220706 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 13:30:52.223762 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.223746 2580 state_mem.go:36] "Initialized new in-memory state store" Sep 16 13:30:52.223882 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.223872 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 13:30:52.223917 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.223914 2580 kubelet.go:486] "Attempting to sync node with API server" Sep 16 13:30:52.223949 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.223929 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 13:30:52.223949 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.223943 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 13:30:52.224005 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.223953 2580 kubelet.go:398] "Adding apiserver pod source" Sep 16 13:30:52.224005 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.223961 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 13:30:52.224537 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.224521 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 13:30:52.224598 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.224546 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 13:30:52.226719 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.226692 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 13:30:52.228874 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.228861 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 13:30:52.228939 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.228892 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 13:30:52.230388 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.230377 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 13:30:52.230433 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.230394 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 13:30:52.230433 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.230399 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 13:30:52.230433 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.230404 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 13:30:52.230433 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.230413 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 13:30:52.230433 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.230418 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 13:30:52.230433 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.230430 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 13:30:52.230433 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.230434 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 13:30:52.230623 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.230442 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 13:30:52.230623 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.230447 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 13:30:52.230623 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.230456 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 13:30:52.230623 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.230461 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 13:30:52.231076 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.231062 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 13:30:52.231123 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.231080 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 13:30:52.235060 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.235037 2580 server.go:1268] "Started kubelet" Sep 16 13:30:52.235145 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.235096 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 13:30:52.238712 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.235199 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 13:30:52.238844 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.238815 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 13:30:52.239874 ip-10-0-132-9 systemd[1]: Started Kubernetes Kubelet. Sep 16 13:30:52.241053 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.239932 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 13:30:52.241338 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:52.241214 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 13:30:52.243195 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.243177 2580 server.go:310] "Adding debug handlers to kubelet server" Sep 16 13:30:52.243471 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.243455 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-9.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 13:30:52.243726 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:52.243705 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 13:30:52.243871 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:52.243852 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-9.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 13:30:52.244264 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:52.243297 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-9.ec2.internal.18d5d04c1ac414ab default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-9.ec2.internal,UID:ip-10-0-132-9.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-9.ec2.internal,},FirstTimestamp:2026-09-16 13:30:52.234790059 +0000 UTC m=+0.335602765,LastTimestamp:2026-09-16 13:30:52.234790059 +0000 UTC m=+0.335602765,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-9.ec2.internal,}" Sep 16 13:30:52.244783 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.244770 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 13:30:52.244842 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.244793 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 13:30:52.244892 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:52.244865 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-9.ec2.internal\" not found" Sep 16 13:30:52.245365 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.245349 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 13:30:52.245365 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.245351 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 13:30:52.245511 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.245375 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 13:30:52.245511 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.245466 2580 reconstruct.go:97] "Volume reconstruction finished" Sep 16 13:30:52.245511 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.245473 2580 reconciler.go:29] "Reconciler: start to sync state" Sep 16 13:30:52.245678 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.245661 2580 factory.go:55] Registering systemd factory Sep 16 13:30:52.245678 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.245677 2580 factory.go:223] Registration of the systemd container factory successfully Sep 16 13:30:52.245898 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.245885 2580 factory.go:153] Registering CRI-O factory Sep 16 13:30:52.245898 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.245900 2580 factory.go:223] Registration of the crio container factory successfully Sep 16 13:30:52.246015 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.245942 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 13:30:52.246015 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.245961 2580 factory.go:103] Registering Raw factory Sep 16 13:30:52.246015 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.245982 2580 manager.go:1196] Started watching for new ooms in manager Sep 16 13:30:52.246429 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.246417 2580 manager.go:349] Starting recovery of all containers Sep 16 13:30:52.247215 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:52.247178 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-9.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 16 13:30:52.247351 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:52.247333 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 16 13:30:52.252636 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.252596 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n6zs8" Sep 16 13:30:52.253842 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.253806 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 13:30:52.257336 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.257319 2580 manager.go:354] Recovery completed Sep 16 13:30:52.262707 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.262696 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:30:52.262992 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.262978 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n6zs8" Sep 16 13:30:52.264767 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.264751 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:30:52.264822 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.264782 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:30:52.264822 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.264795 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:30:52.265428 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.265414 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 13:30:52.265428 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.265429 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 13:30:52.265535 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.265447 2580 state_mem.go:36] "Initialized new in-memory state store" Sep 16 13:30:52.265535 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.265446 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:30:52.266245 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.266229 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:30:52.266308 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.266255 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:30:52.266308 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.266266 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:30:52.266552 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.266532 2580 policy_none.go:49] "None policy: Start" Sep 16 13:30:52.266633 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.266555 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 13:30:52.266633 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.266570 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 13:30:52.267534 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.267524 2580 policy_none.go:47] "Start" Sep 16 13:30:52.316108 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.311672 2580 manager.go:337] "Starting Device Plugin manager" Sep 16 13:30:52.316108 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:52.311712 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 13:30:52.316108 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.311722 2580 server.go:87] "Starting device plugin registration server" Sep 16 13:30:52.316108 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.311967 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 13:30:52.316108 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.311978 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 13:30:52.316108 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.312079 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 13:30:52.316108 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.312187 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 13:30:52.316108 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.312197 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 13:30:52.316108 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:52.312564 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 13:30:52.316108 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:52.312621 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-9.ec2.internal\" not found" Sep 16 13:30:52.348441 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.348415 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 13:30:52.348534 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.348467 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 13:30:52.348534 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.348486 2580 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 13:30:52.348534 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:52.348520 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 13:30:52.350849 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.350829 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:30:52.412678 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.412641 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:30:52.413676 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.413652 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:30:52.413792 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.413686 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:30:52.413792 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.413697 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:30:52.413792 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.413726 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-9.ec2.internal" Sep 16 13:30:52.421656 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.421601 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-9.ec2.internal" Sep 16 13:30:52.421656 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:52.421638 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-9.ec2.internal\": node \"ip-10-0-132-9.ec2.internal\" not found" Sep 16 13:30:52.441754 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:52.441731 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-9.ec2.internal\" not found" Sep 16 13:30:52.449021 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.449001 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-9.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal"] Sep 16 13:30:52.449080 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.449070 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:30:52.450487 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.450472 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:30:52.450564 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.450501 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:30:52.450564 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.450515 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:30:52.451580 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.451566 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:30:52.451726 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.451711 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-9.ec2.internal" Sep 16 13:30:52.451785 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.451747 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:30:52.452261 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.452248 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:30:52.452331 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.452268 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:30:52.452331 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.452275 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:30:52.452331 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.452286 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:30:52.452331 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.452291 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:30:52.452331 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.452295 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:30:52.453258 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.453243 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" Sep 16 13:30:52.453339 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.453267 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:30:52.453955 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.453927 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:30:52.454049 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.453959 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:30:52.454049 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.453969 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:30:52.479260 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:52.479229 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-9.ec2.internal\" not found" node="ip-10-0-132-9.ec2.internal" Sep 16 13:30:52.483515 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:52.483498 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-9.ec2.internal\" not found" node="ip-10-0-132-9.ec2.internal" Sep 16 13:30:52.542825 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:52.542797 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-9.ec2.internal\" not found" Sep 16 13:30:52.547130 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.547112 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4fc4f536cfbc959096a0f31f1c1c0873-config\") pod \"kube-apiserver-proxy-ip-10-0-132-9.ec2.internal\" (UID: \"4fc4f536cfbc959096a0f31f1c1c0873\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-9.ec2.internal" Sep 16 13:30:52.547184 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.547138 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/041a12f64202249206ea5b6ffae99644-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal\" (UID: \"041a12f64202249206ea5b6ffae99644\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" Sep 16 13:30:52.547184 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.547164 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/041a12f64202249206ea5b6ffae99644-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal\" (UID: \"041a12f64202249206ea5b6ffae99644\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" Sep 16 13:30:52.643081 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:52.643046 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-9.ec2.internal\" not found" Sep 16 13:30:52.647393 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.647370 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4fc4f536cfbc959096a0f31f1c1c0873-config\") pod \"kube-apiserver-proxy-ip-10-0-132-9.ec2.internal\" (UID: \"4fc4f536cfbc959096a0f31f1c1c0873\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-9.ec2.internal" Sep 16 13:30:52.647440 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.647410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/041a12f64202249206ea5b6ffae99644-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal\" (UID: \"041a12f64202249206ea5b6ffae99644\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" Sep 16 13:30:52.647440 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.647430 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/041a12f64202249206ea5b6ffae99644-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal\" (UID: \"041a12f64202249206ea5b6ffae99644\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" Sep 16 13:30:52.647499 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.647451 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4fc4f536cfbc959096a0f31f1c1c0873-config\") pod \"kube-apiserver-proxy-ip-10-0-132-9.ec2.internal\" (UID: \"4fc4f536cfbc959096a0f31f1c1c0873\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-9.ec2.internal" Sep 16 13:30:52.647499 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.647470 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/041a12f64202249206ea5b6ffae99644-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal\" (UID: \"041a12f64202249206ea5b6ffae99644\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" Sep 16 13:30:52.647499 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.647472 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/041a12f64202249206ea5b6ffae99644-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal\" (UID: \"041a12f64202249206ea5b6ffae99644\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" Sep 16 13:30:52.743853 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:52.743795 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-9.ec2.internal\" not found" Sep 16 13:30:52.781224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.781192 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-9.ec2.internal" Sep 16 13:30:52.786046 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:52.786030 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" Sep 16 13:30:52.844373 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:52.844339 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-9.ec2.internal\" not found" Sep 16 13:30:52.944802 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:52.944772 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-9.ec2.internal\" not found" Sep 16 13:30:53.045304 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:53.045229 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-9.ec2.internal\" not found" Sep 16 13:30:53.141912 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:53.141881 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:30:53.145944 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:53.145926 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-9.ec2.internal\" not found" Sep 16 13:30:53.183300 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:53.183265 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 13:30:53.183758 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:53.183487 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 13:30:53.183758 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:53.183485 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 13:30:53.242068 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:53.242037 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc4f536cfbc959096a0f31f1c1c0873.slice/crio-0a549813b984ae28c3cea4b775af4b9beae4b575e1179bf73fea47385c31d199 WatchSource:0}: Error finding container 0a549813b984ae28c3cea4b775af4b9beae4b575e1179bf73fea47385c31d199: Status 404 returned error can't find the container with id 0a549813b984ae28c3cea4b775af4b9beae4b575e1179bf73fea47385c31d199 Sep 16 13:30:53.246048 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:53.246027 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-9.ec2.internal\" not found" Sep 16 13:30:53.246796 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:53.246781 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 13:30:53.265963 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:53.265929 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 13:25:52 +0000 UTC" deadline="2028-05-09 01:25:08.171778601 +0000 UTC" Sep 16 13:30:53.265963 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:53.265954 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14411h54m14.90582729s" Sep 16 13:30:53.285661 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:53.285631 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041a12f64202249206ea5b6ffae99644.slice/crio-961289738ea1f83b026386af6ba9925c24df13dae4baffcfc3aa85ff54dc6b19 WatchSource:0}: Error finding container 961289738ea1f83b026386af6ba9925c24df13dae4baffcfc3aa85ff54dc6b19: Status 404 returned error can't find the container with id 961289738ea1f83b026386af6ba9925c24df13dae4baffcfc3aa85ff54dc6b19 Sep 16 13:30:53.347090 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:53.347007 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-9.ec2.internal\" not found" Sep 16 13:30:53.352516 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:53.352474 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" event={"ID":"041a12f64202249206ea5b6ffae99644","Type":"ContainerStarted","Data":"961289738ea1f83b026386af6ba9925c24df13dae4baffcfc3aa85ff54dc6b19"} Sep 16 13:30:53.353561 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:53.353534 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-9.ec2.internal" event={"ID":"4fc4f536cfbc959096a0f31f1c1c0873","Type":"ContainerStarted","Data":"0a549813b984ae28c3cea4b775af4b9beae4b575e1179bf73fea47385c31d199"} Sep 16 13:30:53.447912 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:53.447886 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-9.ec2.internal\" not found" Sep 16 13:30:53.528330 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:53.528298 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:30:53.548007 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:53.547983 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-9.ec2.internal\" not found" Sep 16 13:30:53.583485 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:53.583463 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:30:53.646058 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:53.645972 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-9.ec2.internal" Sep 16 13:30:53.655979 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:53.655955 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 13:30:53.656834 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:53.656821 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" Sep 16 13:30:53.668763 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:53.668739 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 13:30:54.225115 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.224852 2580 apiserver.go:52] "Watching apiserver" Sep 16 13:30:54.235841 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.235813 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 13:30:54.236246 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.236219 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-4wwcp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6","openshift-cluster-node-tuning-operator/tuned-xdbpd","openshift-dns/node-resolver-mf77j","openshift-multus/network-metrics-daemon-c4xxk","openshift-network-operator/iptables-alerter-wdrqt","kube-system/kube-apiserver-proxy-ip-10-0-132-9.ec2.internal","openshift-image-registry/node-ca-mslhp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal","openshift-multus/multus-97dc7","openshift-multus/multus-additional-cni-plugins-wml9m","openshift-network-diagnostics/network-check-target-m6j5f","openshift-ovn-kubernetes/ovnkube-node-4l8qb","kube-system/global-pull-secret-syncer-cmvcd"] Sep 16 13:30:54.237741 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.237706 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.238965 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.238942 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.240177 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.240154 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.241285 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.240713 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 13:30:54.241285 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.240770 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 13:30:54.241285 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.240923 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mfwnd\"" Sep 16 13:30:54.241285 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.241053 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 13:30:54.241285 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.241144 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 13:30:54.241285 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.241232 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mf77j" Sep 16 13:30:54.241732 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.241714 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 13:30:54.241809 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.241750 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 13:30:54.241864 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.241848 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 13:30:54.241916 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.241884 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 13:30:54.241995 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.241977 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r9f9p\"" Sep 16 13:30:54.242550 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.242531 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:30:54.242639 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:54.242609 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4xxk" podUID="cd501395-aaf2-4005-95bd-f202a13d61b5" Sep 16 13:30:54.242773 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.242750 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 13:30:54.242943 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.242922 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 13:30:54.243367 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.243352 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:30:54.243534 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.243515 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 13:30:54.243767 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.243746 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-54zlg\"" Sep 16 13:30:54.243869 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.243836 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wdrqt" Sep 16 13:30:54.244224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.244204 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x2s27\"" Sep 16 13:30:54.244312 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.244299 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 13:30:54.244493 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.244474 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 13:30:54.245029 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.244993 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 13:30:54.245334 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.245310 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:30:54.245407 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:54.245380 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmvcd" podUID="5f469d5f-6b1f-4f0e-910d-bff21be9a372" Sep 16 13:30:54.246346 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.246325 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mslhp" Sep 16 13:30:54.246975 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.246785 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:30:54.247170 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.247148 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mxmlk\"" Sep 16 13:30:54.247267 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.247179 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 13:30:54.247335 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.247289 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 13:30:54.247881 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.247855 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.249236 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.249214 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4wwcp" Sep 16 13:30:54.250125 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.250102 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 13:30:54.250213 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.250105 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 13:30:54.251221 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.251202 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.252395 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.252375 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 13:30:54.252706 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.252656 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-68mtn\"" Sep 16 13:30:54.252797 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.252713 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 13:30:54.252797 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.252754 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n47v5\"" Sep 16 13:30:54.253037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.253019 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 13:30:54.253242 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.253145 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 13:30:54.253242 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.253159 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:30:54.253242 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:54.253231 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6j5f" podUID="b313af30-1af8-4dac-a6f6-fa42450ea3df" Sep 16 13:30:54.253885 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.253862 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 13:30:54.254668 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.254634 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 13:30:54.254668 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.254647 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 13:30:54.254876 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.254860 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bnr5g\"" Sep 16 13:30:54.255094 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.255074 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 13:30:54.255170 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.255132 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-prnkq\"" Sep 16 13:30:54.256230 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256212 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b1c10dc-eefa-4040-91e6-c52ce36a7ac5-host\") pod \"node-ca-mslhp\" (UID: \"3b1c10dc-eefa-4040-91e6-c52ce36a7ac5\") " pod="openshift-image-registry/node-ca-mslhp" Sep 16 13:30:54.256330 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256240 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b1c10dc-eefa-4040-91e6-c52ce36a7ac5-serviceca\") pod \"node-ca-mslhp\" (UID: \"3b1c10dc-eefa-4040-91e6-c52ce36a7ac5\") " pod="openshift-image-registry/node-ca-mslhp" Sep 16 13:30:54.256330 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256274 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-host-run-netns\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.256455 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256334 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-host-kubelet\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.256455 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256372 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-node-log\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.256455 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256396 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-host-run-ovn-kubernetes\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.256455 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256421 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-ovnkube-script-lib\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.256455 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256444 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-etc-sysctl-conf\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.256788 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256468 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-lib-modules\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.256788 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256495 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-host-cni-netd\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.256788 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256537 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-run\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.256788 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256570 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-etc-sysctl-d\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.256788 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256592 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-cnibin\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.256788 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256629 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-systemd-units\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.256788 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256653 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-ovnkube-config\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.256788 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256701 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-var-lib-kubelet\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.256788 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256741 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-host\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.256788 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256763 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-tmp\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.257290 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256795 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50b34db9-0ba0-4a12-8366-e9bca293d05c-host-slash\") pod \"iptables-alerter-wdrqt\" (UID: \"50b34db9-0ba0-4a12-8366-e9bca293d05c\") " pod="openshift-network-operator/iptables-alerter-wdrqt" Sep 16 13:30:54.257290 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256843 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-system-cni-dir\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.257290 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256877 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-host-run-k8s-cni-cncf-io\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.257290 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256902 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-multus-conf-dir\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.257290 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.256939 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-host-run-multus-certs\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.257290 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.257033 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n28t7\" (UniqueName: \"kubernetes.io/projected/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-kube-api-access-n28t7\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.257290 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.257059 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-multus-cni-dir\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.257290 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.257082 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-multus-socket-dir-parent\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.257290 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.257127 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-host-var-lib-cni-multus\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.257290 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.257158 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-multus-daemon-config\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.257290 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.257183 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-host-run-netns\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.257290 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.257249 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-run-systemd\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.257900 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.257310 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-host-var-lib-cni-bin\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.257900 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.257340 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-host-var-lib-kubelet\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.257900 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.257363 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-sys\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.257900 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.257569 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret\") pod \"global-pull-secret-syncer-cmvcd\" (UID: \"5f469d5f-6b1f-4f0e-910d-bff21be9a372\") " pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:30:54.257900 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.257602 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-host-slash\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.258155 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.257950 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-etc-openvswitch\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.258155 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.258092 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-ovn-node-metrics-cert\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.258254 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.258158 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jtbl\" (UniqueName: \"kubernetes.io/projected/3b1c10dc-eefa-4040-91e6-c52ce36a7ac5-kube-api-access-2jtbl\") pod \"node-ca-mslhp\" (UID: \"3b1c10dc-eefa-4040-91e6-c52ce36a7ac5\") " pod="openshift-image-registry/node-ca-mslhp" Sep 16 13:30:54.258254 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.258210 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab189943-4b62-4e77-87bd-75934dcb19cb-tmp-dir\") pod \"node-resolver-mf77j\" (UID: \"ab189943-4b62-4e77-87bd-75934dcb19cb\") " pod="openshift-dns/node-resolver-mf77j" Sep 16 13:30:54.258355 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.258258 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-etc-sysconfig\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.258355 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.258311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-etc-systemd\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.258453 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.258366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-cni-binary-copy\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.258649 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.258573 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnt7j\" (UniqueName: \"kubernetes.io/projected/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-kube-api-access-jnt7j\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.259019 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.258992 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 13:30:54.259363 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.258753 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f469d5f-6b1f-4f0e-910d-bff21be9a372-kubelet-config\") pod \"global-pull-secret-syncer-cmvcd\" (UID: \"5f469d5f-6b1f-4f0e-910d-bff21be9a372\") " pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:30:54.259438 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.259406 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-hostroot\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.259505 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.259459 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-env-overrides\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.259562 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.259501 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlgbn\" (UniqueName: \"kubernetes.io/projected/50b34db9-0ba0-4a12-8366-e9bca293d05c-kube-api-access-mlgbn\") pod \"iptables-alerter-wdrqt\" (UID: \"50b34db9-0ba0-4a12-8366-e9bca293d05c\") " pod="openshift-network-operator/iptables-alerter-wdrqt" Sep 16 13:30:54.259618 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.259579 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.259700 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.259631 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fw4w\" (UniqueName: \"kubernetes.io/projected/cd501395-aaf2-4005-95bd-f202a13d61b5-kube-api-access-7fw4w\") pod \"network-metrics-daemon-c4xxk\" (UID: \"cd501395-aaf2-4005-95bd-f202a13d61b5\") " pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:30:54.259700 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.259663 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-run-openvswitch\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.259808 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.259718 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-host-cni-bin\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.259808 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.259752 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-etc-modprobe-d\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.259808 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.259777 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-etc-kubernetes\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.259808 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.259797 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx9ht\" (UniqueName: \"kubernetes.io/projected/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-kube-api-access-mx9ht\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.260004 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.259816 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v46kw\" (UniqueName: \"kubernetes.io/projected/ab189943-4b62-4e77-87bd-75934dcb19cb-kube-api-access-v46kw\") pod \"node-resolver-mf77j\" (UID: \"ab189943-4b62-4e77-87bd-75934dcb19cb\") " pod="openshift-dns/node-resolver-mf77j" Sep 16 13:30:54.260004 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.259834 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-etc-tuned\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.260004 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.259854 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-run-ovn\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.260004 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.259872 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-log-socket\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.260004 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.259888 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-etc-kubernetes\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.260004 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.259907 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f469d5f-6b1f-4f0e-910d-bff21be9a372-dbus\") pod \"global-pull-secret-syncer-cmvcd\" (UID: \"5f469d5f-6b1f-4f0e-910d-bff21be9a372\") " pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:30:54.260004 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.259926 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs\") pod \"network-metrics-daemon-c4xxk\" (UID: \"cd501395-aaf2-4005-95bd-f202a13d61b5\") " pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:30:54.260004 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.259947 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/50b34db9-0ba0-4a12-8366-e9bca293d05c-iptables-alerter-script\") pod \"iptables-alerter-wdrqt\" (UID: \"50b34db9-0ba0-4a12-8366-e9bca293d05c\") " pod="openshift-network-operator/iptables-alerter-wdrqt" Sep 16 13:30:54.260004 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.259967 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-var-lib-openvswitch\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.260004 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.259983 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab189943-4b62-4e77-87bd-75934dcb19cb-hosts-file\") pod \"node-resolver-mf77j\" (UID: \"ab189943-4b62-4e77-87bd-75934dcb19cb\") " pod="openshift-dns/node-resolver-mf77j" Sep 16 13:30:54.285017 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.284988 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dls26" Sep 16 13:30:54.293386 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.293361 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dls26" Sep 16 13:30:54.346854 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.346822 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 13:30:54.360809 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.360779 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlgbn\" (UniqueName: \"kubernetes.io/projected/50b34db9-0ba0-4a12-8366-e9bca293d05c-kube-api-access-mlgbn\") pod \"iptables-alerter-wdrqt\" (UID: \"50b34db9-0ba0-4a12-8366-e9bca293d05c\") " pod="openshift-network-operator/iptables-alerter-wdrqt" Sep 16 13:30:54.360995 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.360820 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.360995 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.360853 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fw4w\" (UniqueName: \"kubernetes.io/projected/cd501395-aaf2-4005-95bd-f202a13d61b5-kube-api-access-7fw4w\") pod \"network-metrics-daemon-c4xxk\" (UID: \"cd501395-aaf2-4005-95bd-f202a13d61b5\") " pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:30:54.360995 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.360882 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5210d309-d43d-4468-861b-7d2a3ac8b4d5-registration-dir\") pod \"aws-ebs-csi-driver-node-q5jv6\" (UID: \"5210d309-d43d-4468-861b-7d2a3ac8b4d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.360995 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.360907 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wjpw\" (UniqueName: \"kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw\") pod \"network-check-target-m6j5f\" (UID: \"b313af30-1af8-4dac-a6f6-fa42450ea3df\") " pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:30:54.360995 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.360934 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-run-openvswitch\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.360995 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.360958 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.360995 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.360963 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-host-cni-bin\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.361372 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361009 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-host-cni-bin\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.361372 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361008 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-etc-modprobe-d\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.361372 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361037 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-etc-kubernetes\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.361372 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361056 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx9ht\" (UniqueName: \"kubernetes.io/projected/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-kube-api-access-mx9ht\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.361372 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361083 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c53ff24-05da-4f65-8508-84dda5a72a7d-konnectivity-ca\") pod \"konnectivity-agent-4wwcp\" (UID: \"4c53ff24-05da-4f65-8508-84dda5a72a7d\") " pod="kube-system/konnectivity-agent-4wwcp" Sep 16 13:30:54.361372 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361100 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v46kw\" (UniqueName: \"kubernetes.io/projected/ab189943-4b62-4e77-87bd-75934dcb19cb-kube-api-access-v46kw\") pod \"node-resolver-mf77j\" (UID: \"ab189943-4b62-4e77-87bd-75934dcb19cb\") " pod="openshift-dns/node-resolver-mf77j" Sep 16 13:30:54.361372 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361118 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5210d309-d43d-4468-861b-7d2a3ac8b4d5-sys-fs\") pod \"aws-ebs-csi-driver-node-q5jv6\" (UID: \"5210d309-d43d-4468-861b-7d2a3ac8b4d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.361372 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361121 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-etc-modprobe-d\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.361372 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361138 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-etc-tuned\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.361372 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361166 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-run-ovn\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.361372 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361183 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/70b7167e-0042-4b50-905c-2cb444c08bc3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wml9m\" (UID: \"70b7167e-0042-4b50-905c-2cb444c08bc3\") " pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.361372 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361189 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-run-openvswitch\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.361372 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361205 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-log-socket\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.361372 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361251 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-run-ovn\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.361372 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361261 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-etc-kubernetes\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.361372 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361288 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-etc-kubernetes\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.361372 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f469d5f-6b1f-4f0e-910d-bff21be9a372-dbus\") pod \"global-pull-secret-syncer-cmvcd\" (UID: \"5f469d5f-6b1f-4f0e-910d-bff21be9a372\") " pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:30:54.362151 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361305 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-log-socket\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.362151 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361336 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs\") pod \"network-metrics-daemon-c4xxk\" (UID: \"cd501395-aaf2-4005-95bd-f202a13d61b5\") " pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:30:54.362151 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361398 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-etc-kubernetes\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.362151 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361393 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/70b7167e-0042-4b50-905c-2cb444c08bc3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wml9m\" (UID: \"70b7167e-0042-4b50-905c-2cb444c08bc3\") " pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.362151 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:54.361424 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:30:54.362151 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/50b34db9-0ba0-4a12-8366-e9bca293d05c-iptables-alerter-script\") pod \"iptables-alerter-wdrqt\" (UID: \"50b34db9-0ba0-4a12-8366-e9bca293d05c\") " pod="openshift-network-operator/iptables-alerter-wdrqt" Sep 16 13:30:54.362151 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361522 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-var-lib-openvswitch\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.362151 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361631 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-var-lib-openvswitch\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.362151 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:54.361638 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs podName:cd501395-aaf2-4005-95bd-f202a13d61b5 nodeName:}" failed. No retries permitted until 2026-09-16 13:30:54.861580071 +0000 UTC m=+2.962392795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs") pod "network-metrics-daemon-c4xxk" (UID: "cd501395-aaf2-4005-95bd-f202a13d61b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:30:54.362151 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361703 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab189943-4b62-4e77-87bd-75934dcb19cb-hosts-file\") pod \"node-resolver-mf77j\" (UID: \"ab189943-4b62-4e77-87bd-75934dcb19cb\") " pod="openshift-dns/node-resolver-mf77j" Sep 16 13:30:54.362151 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361734 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c53ff24-05da-4f65-8508-84dda5a72a7d-agent-certs\") pod \"konnectivity-agent-4wwcp\" (UID: \"4c53ff24-05da-4f65-8508-84dda5a72a7d\") " pod="kube-system/konnectivity-agent-4wwcp" Sep 16 13:30:54.362151 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361720 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 13:30:54.362151 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361768 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b1c10dc-eefa-4040-91e6-c52ce36a7ac5-host\") pod \"node-ca-mslhp\" (UID: \"3b1c10dc-eefa-4040-91e6-c52ce36a7ac5\") " pod="openshift-image-registry/node-ca-mslhp" Sep 16 13:30:54.362151 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361790 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b1c10dc-eefa-4040-91e6-c52ce36a7ac5-serviceca\") pod \"node-ca-mslhp\" (UID: \"3b1c10dc-eefa-4040-91e6-c52ce36a7ac5\") " pod="openshift-image-registry/node-ca-mslhp" Sep 16 13:30:54.362151 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361795 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f469d5f-6b1f-4f0e-910d-bff21be9a372-dbus\") pod \"global-pull-secret-syncer-cmvcd\" (UID: \"5f469d5f-6b1f-4f0e-910d-bff21be9a372\") " pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:30:54.362151 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361810 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-host-run-netns\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.362151 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361835 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-host-kubelet\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.363101 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361847 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b1c10dc-eefa-4040-91e6-c52ce36a7ac5-host\") pod \"node-ca-mslhp\" (UID: \"3b1c10dc-eefa-4040-91e6-c52ce36a7ac5\") " pod="openshift-image-registry/node-ca-mslhp" Sep 16 13:30:54.363101 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361862 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-node-log\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.363101 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361873 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-host-run-netns\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.363101 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361809 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab189943-4b62-4e77-87bd-75934dcb19cb-hosts-file\") pod \"node-resolver-mf77j\" (UID: \"ab189943-4b62-4e77-87bd-75934dcb19cb\") " pod="openshift-dns/node-resolver-mf77j" Sep 16 13:30:54.363101 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361901 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-host-kubelet\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.363101 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361905 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-host-run-ovn-kubernetes\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.363101 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361949 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-node-log\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.363101 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361965 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-ovnkube-script-lib\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.363101 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-etc-sysctl-conf\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.363101 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362013 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-lib-modules\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.363101 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-host-cni-netd\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.363101 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362094 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5210d309-d43d-4468-861b-7d2a3ac8b4d5-device-dir\") pod \"aws-ebs-csi-driver-node-q5jv6\" (UID: \"5210d309-d43d-4468-861b-7d2a3ac8b4d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.363101 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362117 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5210d309-d43d-4468-861b-7d2a3ac8b4d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-q5jv6\" (UID: \"5210d309-d43d-4468-861b-7d2a3ac8b4d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.363101 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362144 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-etc-sysctl-conf\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.363101 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362154 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70b7167e-0042-4b50-905c-2cb444c08bc3-cnibin\") pod \"multus-additional-cni-plugins-wml9m\" (UID: \"70b7167e-0042-4b50-905c-2cb444c08bc3\") " pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.363101 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.361993 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-host-run-ovn-kubernetes\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.363101 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362173 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-lib-modules\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.363887 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362178 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70b7167e-0042-4b50-905c-2cb444c08bc3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wml9m\" (UID: \"70b7167e-0042-4b50-905c-2cb444c08bc3\") " pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.363887 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362212 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-run\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.363887 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362219 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-host-cni-netd\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.363887 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362238 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/50b34db9-0ba0-4a12-8366-e9bca293d05c-iptables-alerter-script\") pod \"iptables-alerter-wdrqt\" (UID: \"50b34db9-0ba0-4a12-8366-e9bca293d05c\") " pod="openshift-network-operator/iptables-alerter-wdrqt" Sep 16 13:30:54.363887 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362254 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-etc-sysctl-d\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.363887 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362263 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-run\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.363887 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362269 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b1c10dc-eefa-4040-91e6-c52ce36a7ac5-serviceca\") pod \"node-ca-mslhp\" (UID: \"3b1c10dc-eefa-4040-91e6-c52ce36a7ac5\") " pod="openshift-image-registry/node-ca-mslhp" Sep 16 13:30:54.363887 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362308 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-cnibin\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.363887 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362357 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-etc-sysctl-d\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.363887 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362382 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-systemd-units\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.363887 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362418 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-cnibin\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.363887 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362473 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-ovnkube-config\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.363887 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362482 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-systemd-units\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.363887 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362520 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-var-lib-kubelet\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.363887 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362608 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-var-lib-kubelet\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.363887 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362625 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-host\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.363887 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362699 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-ovnkube-script-lib\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.363887 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362717 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-host\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.364821 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362704 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-tmp\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.364821 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362763 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50b34db9-0ba0-4a12-8366-e9bca293d05c-host-slash\") pod \"iptables-alerter-wdrqt\" (UID: \"50b34db9-0ba0-4a12-8366-e9bca293d05c\") " pod="openshift-network-operator/iptables-alerter-wdrqt" Sep 16 13:30:54.364821 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362809 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-system-cni-dir\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.364821 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362837 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-host-run-k8s-cni-cncf-io\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.364821 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362861 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50b34db9-0ba0-4a12-8366-e9bca293d05c-host-slash\") pod \"iptables-alerter-wdrqt\" (UID: \"50b34db9-0ba0-4a12-8366-e9bca293d05c\") " pod="openshift-network-operator/iptables-alerter-wdrqt" Sep 16 13:30:54.364821 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362878 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-system-cni-dir\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.364821 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362867 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-multus-conf-dir\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.364821 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362933 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-host-run-k8s-cni-cncf-io\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.364821 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362911 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-multus-conf-dir\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.364821 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-host-run-multus-certs\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.364821 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362980 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n28t7\" (UniqueName: \"kubernetes.io/projected/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-kube-api-access-n28t7\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.364821 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.362980 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-host-run-multus-certs\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.364821 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363008 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-multus-cni-dir\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.364821 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363033 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-multus-socket-dir-parent\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.364821 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363062 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-host-var-lib-cni-multus\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.364821 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363068 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-ovnkube-config\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.364821 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363080 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-multus-socket-dir-parent\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.364821 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363087 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-multus-cni-dir\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.365605 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363090 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-multus-daemon-config\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.365605 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363118 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-host-run-netns\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.365605 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363156 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-host-run-netns\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.365605 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363190 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-host-var-lib-cni-multus\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.365605 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363196 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-run-systemd\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.365605 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363252 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5210d309-d43d-4468-861b-7d2a3ac8b4d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q5jv6\" (UID: \"5210d309-d43d-4468-861b-7d2a3ac8b4d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.365605 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363257 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-run-systemd\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.365605 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363290 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-host-var-lib-cni-bin\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.365605 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363316 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-host-var-lib-kubelet\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.365605 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363356 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-host-var-lib-kubelet\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.365605 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363363 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-host-var-lib-cni-bin\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.365605 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363395 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5210d309-d43d-4468-861b-7d2a3ac8b4d5-socket-dir\") pod \"aws-ebs-csi-driver-node-q5jv6\" (UID: \"5210d309-d43d-4468-861b-7d2a3ac8b4d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.365605 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363425 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8472\" (UniqueName: \"kubernetes.io/projected/5210d309-d43d-4468-861b-7d2a3ac8b4d5-kube-api-access-q8472\") pod \"aws-ebs-csi-driver-node-q5jv6\" (UID: \"5210d309-d43d-4468-861b-7d2a3ac8b4d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.365605 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363463 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-sys\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.365605 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363503 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret\") pod \"global-pull-secret-syncer-cmvcd\" (UID: \"5f469d5f-6b1f-4f0e-910d-bff21be9a372\") " pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:30:54.365605 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363559 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-sys\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.365605 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363629 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-host-slash\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.366432 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363656 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-etc-openvswitch\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.366432 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:54.363676 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:30:54.366432 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363749 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-host-slash\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.366432 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363759 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-etc-openvswitch\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.366432 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363793 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-multus-daemon-config\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.366432 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363681 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-ovn-node-metrics-cert\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.366432 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:54.363841 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret podName:5f469d5f-6b1f-4f0e-910d-bff21be9a372 nodeName:}" failed. No retries permitted until 2026-09-16 13:30:54.863810707 +0000 UTC m=+2.964623419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret") pod "global-pull-secret-syncer-cmvcd" (UID: "5f469d5f-6b1f-4f0e-910d-bff21be9a372") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:30:54.366432 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363894 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jtbl\" (UniqueName: \"kubernetes.io/projected/3b1c10dc-eefa-4040-91e6-c52ce36a7ac5-kube-api-access-2jtbl\") pod \"node-ca-mslhp\" (UID: \"3b1c10dc-eefa-4040-91e6-c52ce36a7ac5\") " pod="openshift-image-registry/node-ca-mslhp" Sep 16 13:30:54.366432 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363923 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab189943-4b62-4e77-87bd-75934dcb19cb-tmp-dir\") pod \"node-resolver-mf77j\" (UID: \"ab189943-4b62-4e77-87bd-75934dcb19cb\") " pod="openshift-dns/node-resolver-mf77j" Sep 16 13:30:54.366432 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.363968 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-etc-sysconfig\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.366432 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.364024 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-etc-systemd\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.366432 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.364117 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-cni-binary-copy\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.366432 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.364148 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnt7j\" (UniqueName: \"kubernetes.io/projected/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-kube-api-access-jnt7j\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.366432 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.364196 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70b7167e-0042-4b50-905c-2cb444c08bc3-system-cni-dir\") pod \"multus-additional-cni-plugins-wml9m\" (UID: \"70b7167e-0042-4b50-905c-2cb444c08bc3\") " pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.366432 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.364222 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70b7167e-0042-4b50-905c-2cb444c08bc3-cni-binary-copy\") pod \"multus-additional-cni-plugins-wml9m\" (UID: \"70b7167e-0042-4b50-905c-2cb444c08bc3\") " pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.366432 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.364245 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-etc-sysconfig\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.366432 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.364251 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp49d\" (UniqueName: \"kubernetes.io/projected/70b7167e-0042-4b50-905c-2cb444c08bc3-kube-api-access-sp49d\") pod \"multus-additional-cni-plugins-wml9m\" (UID: \"70b7167e-0042-4b50-905c-2cb444c08bc3\") " pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.367192 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.364272 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab189943-4b62-4e77-87bd-75934dcb19cb-tmp-dir\") pod \"node-resolver-mf77j\" (UID: \"ab189943-4b62-4e77-87bd-75934dcb19cb\") " pod="openshift-dns/node-resolver-mf77j" Sep 16 13:30:54.367192 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.364318 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f469d5f-6b1f-4f0e-910d-bff21be9a372-kubelet-config\") pod \"global-pull-secret-syncer-cmvcd\" (UID: \"5f469d5f-6b1f-4f0e-910d-bff21be9a372\") " pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:30:54.367192 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.364352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-hostroot\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.367192 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.364381 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-env-overrides\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.367192 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.364387 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-etc-systemd\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.367192 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.364642 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f469d5f-6b1f-4f0e-910d-bff21be9a372-kubelet-config\") pod \"global-pull-secret-syncer-cmvcd\" (UID: \"5f469d5f-6b1f-4f0e-910d-bff21be9a372\") " pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:30:54.367192 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.364732 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-hostroot\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.367192 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.364856 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-env-overrides\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.367192 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.365121 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-cni-binary-copy\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.367192 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.366215 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-etc-tuned\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.367192 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.366367 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-tmp\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.367192 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.366889 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-ovn-node-metrics-cert\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.374538 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.374460 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fw4w\" (UniqueName: \"kubernetes.io/projected/cd501395-aaf2-4005-95bd-f202a13d61b5-kube-api-access-7fw4w\") pod \"network-metrics-daemon-c4xxk\" (UID: \"cd501395-aaf2-4005-95bd-f202a13d61b5\") " pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:30:54.375037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.375006 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlgbn\" (UniqueName: \"kubernetes.io/projected/50b34db9-0ba0-4a12-8366-e9bca293d05c-kube-api-access-mlgbn\") pod \"iptables-alerter-wdrqt\" (UID: \"50b34db9-0ba0-4a12-8366-e9bca293d05c\") " pod="openshift-network-operator/iptables-alerter-wdrqt" Sep 16 13:30:54.375210 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.375192 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v46kw\" (UniqueName: \"kubernetes.io/projected/ab189943-4b62-4e77-87bd-75934dcb19cb-kube-api-access-v46kw\") pod \"node-resolver-mf77j\" (UID: \"ab189943-4b62-4e77-87bd-75934dcb19cb\") " pod="openshift-dns/node-resolver-mf77j" Sep 16 13:30:54.376425 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.376399 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n28t7\" (UniqueName: \"kubernetes.io/projected/7b3b171d-e589-4e4a-a6f4-e87bc11c6207-kube-api-access-n28t7\") pod \"tuned-xdbpd\" (UID: \"7b3b171d-e589-4e4a-a6f4-e87bc11c6207\") " pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.376919 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.376894 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx9ht\" (UniqueName: \"kubernetes.io/projected/d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2-kube-api-access-mx9ht\") pod \"multus-97dc7\" (UID: \"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2\") " pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.377028 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.376972 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnt7j\" (UniqueName: \"kubernetes.io/projected/f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5-kube-api-access-jnt7j\") pod \"ovnkube-node-4l8qb\" (UID: \"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.377704 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.377680 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jtbl\" (UniqueName: \"kubernetes.io/projected/3b1c10dc-eefa-4040-91e6-c52ce36a7ac5-kube-api-access-2jtbl\") pod \"node-ca-mslhp\" (UID: \"3b1c10dc-eefa-4040-91e6-c52ce36a7ac5\") " pod="openshift-image-registry/node-ca-mslhp" Sep 16 13:30:54.396766 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.396742 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:30:54.465109 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5210d309-d43d-4468-861b-7d2a3ac8b4d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q5jv6\" (UID: \"5210d309-d43d-4468-861b-7d2a3ac8b4d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.465109 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465114 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5210d309-d43d-4468-861b-7d2a3ac8b4d5-socket-dir\") pod \"aws-ebs-csi-driver-node-q5jv6\" (UID: \"5210d309-d43d-4468-861b-7d2a3ac8b4d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.465360 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465141 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8472\" (UniqueName: \"kubernetes.io/projected/5210d309-d43d-4468-861b-7d2a3ac8b4d5-kube-api-access-q8472\") pod \"aws-ebs-csi-driver-node-q5jv6\" (UID: \"5210d309-d43d-4468-861b-7d2a3ac8b4d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.465360 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465193 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70b7167e-0042-4b50-905c-2cb444c08bc3-system-cni-dir\") pod \"multus-additional-cni-plugins-wml9m\" (UID: \"70b7167e-0042-4b50-905c-2cb444c08bc3\") " pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.465360 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465217 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70b7167e-0042-4b50-905c-2cb444c08bc3-cni-binary-copy\") pod \"multus-additional-cni-plugins-wml9m\" (UID: \"70b7167e-0042-4b50-905c-2cb444c08bc3\") " pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.465360 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465221 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5210d309-d43d-4468-861b-7d2a3ac8b4d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q5jv6\" (UID: \"5210d309-d43d-4468-861b-7d2a3ac8b4d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.465360 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465240 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp49d\" (UniqueName: \"kubernetes.io/projected/70b7167e-0042-4b50-905c-2cb444c08bc3-kube-api-access-sp49d\") pod \"multus-additional-cni-plugins-wml9m\" (UID: \"70b7167e-0042-4b50-905c-2cb444c08bc3\") " pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.465360 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465294 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5210d309-d43d-4468-861b-7d2a3ac8b4d5-registration-dir\") pod \"aws-ebs-csi-driver-node-q5jv6\" (UID: \"5210d309-d43d-4468-861b-7d2a3ac8b4d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.465360 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465300 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70b7167e-0042-4b50-905c-2cb444c08bc3-system-cni-dir\") pod \"multus-additional-cni-plugins-wml9m\" (UID: \"70b7167e-0042-4b50-905c-2cb444c08bc3\") " pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.465360 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465324 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjpw\" (UniqueName: \"kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw\") pod \"network-check-target-m6j5f\" (UID: \"b313af30-1af8-4dac-a6f6-fa42450ea3df\") " pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:30:54.465360 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465306 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5210d309-d43d-4468-861b-7d2a3ac8b4d5-socket-dir\") pod \"aws-ebs-csi-driver-node-q5jv6\" (UID: \"5210d309-d43d-4468-861b-7d2a3ac8b4d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.465360 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465360 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c53ff24-05da-4f65-8508-84dda5a72a7d-konnectivity-ca\") pod \"konnectivity-agent-4wwcp\" (UID: \"4c53ff24-05da-4f65-8508-84dda5a72a7d\") " pod="kube-system/konnectivity-agent-4wwcp" Sep 16 13:30:54.465884 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465389 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5210d309-d43d-4468-861b-7d2a3ac8b4d5-sys-fs\") pod \"aws-ebs-csi-driver-node-q5jv6\" (UID: \"5210d309-d43d-4468-861b-7d2a3ac8b4d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.465884 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465413 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/70b7167e-0042-4b50-905c-2cb444c08bc3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wml9m\" (UID: \"70b7167e-0042-4b50-905c-2cb444c08bc3\") " pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.465884 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465441 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/70b7167e-0042-4b50-905c-2cb444c08bc3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wml9m\" (UID: \"70b7167e-0042-4b50-905c-2cb444c08bc3\") " pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.465884 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465463 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c53ff24-05da-4f65-8508-84dda5a72a7d-agent-certs\") pod \"konnectivity-agent-4wwcp\" (UID: \"4c53ff24-05da-4f65-8508-84dda5a72a7d\") " pod="kube-system/konnectivity-agent-4wwcp" Sep 16 13:30:54.465884 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465383 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5210d309-d43d-4468-861b-7d2a3ac8b4d5-registration-dir\") pod \"aws-ebs-csi-driver-node-q5jv6\" (UID: \"5210d309-d43d-4468-861b-7d2a3ac8b4d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.465884 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465502 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5210d309-d43d-4468-861b-7d2a3ac8b4d5-device-dir\") pod \"aws-ebs-csi-driver-node-q5jv6\" (UID: \"5210d309-d43d-4468-861b-7d2a3ac8b4d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.465884 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5210d309-d43d-4468-861b-7d2a3ac8b4d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-q5jv6\" (UID: \"5210d309-d43d-4468-861b-7d2a3ac8b4d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.465884 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465532 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70b7167e-0042-4b50-905c-2cb444c08bc3-cnibin\") pod \"multus-additional-cni-plugins-wml9m\" (UID: \"70b7167e-0042-4b50-905c-2cb444c08bc3\") " pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.465884 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465555 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5210d309-d43d-4468-861b-7d2a3ac8b4d5-sys-fs\") pod \"aws-ebs-csi-driver-node-q5jv6\" (UID: \"5210d309-d43d-4468-861b-7d2a3ac8b4d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.465884 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465551 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70b7167e-0042-4b50-905c-2cb444c08bc3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wml9m\" (UID: \"70b7167e-0042-4b50-905c-2cb444c08bc3\") " pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.465884 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465729 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5210d309-d43d-4468-861b-7d2a3ac8b4d5-device-dir\") pod \"aws-ebs-csi-driver-node-q5jv6\" (UID: \"5210d309-d43d-4468-861b-7d2a3ac8b4d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.466405 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.465917 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70b7167e-0042-4b50-905c-2cb444c08bc3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wml9m\" (UID: \"70b7167e-0042-4b50-905c-2cb444c08bc3\") " pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.466405 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.466034 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5210d309-d43d-4468-861b-7d2a3ac8b4d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-q5jv6\" (UID: \"5210d309-d43d-4468-861b-7d2a3ac8b4d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.466405 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.466067 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/70b7167e-0042-4b50-905c-2cb444c08bc3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wml9m\" (UID: \"70b7167e-0042-4b50-905c-2cb444c08bc3\") " pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.466405 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.466084 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70b7167e-0042-4b50-905c-2cb444c08bc3-cnibin\") pod \"multus-additional-cni-plugins-wml9m\" (UID: \"70b7167e-0042-4b50-905c-2cb444c08bc3\") " pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.466405 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.466094 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c53ff24-05da-4f65-8508-84dda5a72a7d-konnectivity-ca\") pod \"konnectivity-agent-4wwcp\" (UID: \"4c53ff24-05da-4f65-8508-84dda5a72a7d\") " pod="kube-system/konnectivity-agent-4wwcp" Sep 16 13:30:54.466405 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.466095 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70b7167e-0042-4b50-905c-2cb444c08bc3-cni-binary-copy\") pod \"multus-additional-cni-plugins-wml9m\" (UID: \"70b7167e-0042-4b50-905c-2cb444c08bc3\") " pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.466405 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.466173 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/70b7167e-0042-4b50-905c-2cb444c08bc3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wml9m\" (UID: \"70b7167e-0042-4b50-905c-2cb444c08bc3\") " pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.468625 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.468583 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c53ff24-05da-4f65-8508-84dda5a72a7d-agent-certs\") pod \"konnectivity-agent-4wwcp\" (UID: \"4c53ff24-05da-4f65-8508-84dda5a72a7d\") " pod="kube-system/konnectivity-agent-4wwcp" Sep 16 13:30:54.472744 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:54.472718 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:30:54.472744 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:54.472744 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:30:54.472927 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:54.472758 2580 projected.go:196] Error preparing data for projected volume kube-api-access-2wjpw for pod openshift-network-diagnostics/network-check-target-m6j5f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:30:54.472927 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:54.472868 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw podName:b313af30-1af8-4dac-a6f6-fa42450ea3df nodeName:}" failed. No retries permitted until 2026-09-16 13:30:54.972850023 +0000 UTC m=+3.073662732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2wjpw" (UniqueName: "kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw") pod "network-check-target-m6j5f" (UID: "b313af30-1af8-4dac-a6f6-fa42450ea3df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:30:54.475154 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.475100 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp49d\" (UniqueName: \"kubernetes.io/projected/70b7167e-0042-4b50-905c-2cb444c08bc3-kube-api-access-sp49d\") pod \"multus-additional-cni-plugins-wml9m\" (UID: \"70b7167e-0042-4b50-905c-2cb444c08bc3\") " pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.475453 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.475434 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8472\" (UniqueName: \"kubernetes.io/projected/5210d309-d43d-4468-861b-7d2a3ac8b4d5-kube-api-access-q8472\") pod \"aws-ebs-csi-driver-node-q5jv6\" (UID: \"5210d309-d43d-4468-861b-7d2a3ac8b4d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.551159 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.551115 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-97dc7" Sep 16 13:30:54.561216 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.561189 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:30:54.572099 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.572072 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" Sep 16 13:30:54.576739 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.576717 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mf77j" Sep 16 13:30:54.587703 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.587666 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wdrqt" Sep 16 13:30:54.595458 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.595426 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mslhp" Sep 16 13:30:54.603966 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.603943 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" Sep 16 13:30:54.612139 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.612113 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4wwcp" Sep 16 13:30:54.618861 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.618840 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wml9m" Sep 16 13:30:54.868991 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.868885 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret\") pod \"global-pull-secret-syncer-cmvcd\" (UID: \"5f469d5f-6b1f-4f0e-910d-bff21be9a372\") " pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:30:54.868991 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.868974 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs\") pod \"network-metrics-daemon-c4xxk\" (UID: \"cd501395-aaf2-4005-95bd-f202a13d61b5\") " pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:30:54.869213 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:54.869016 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:30:54.869213 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:54.869076 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:30:54.869213 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:54.869084 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret podName:5f469d5f-6b1f-4f0e-910d-bff21be9a372 nodeName:}" failed. No retries permitted until 2026-09-16 13:30:55.869069283 +0000 UTC m=+3.969881992 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret") pod "global-pull-secret-syncer-cmvcd" (UID: "5f469d5f-6b1f-4f0e-910d-bff21be9a372") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:30:54.869213 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:54.869125 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs podName:cd501395-aaf2-4005-95bd-f202a13d61b5 nodeName:}" failed. No retries permitted until 2026-09-16 13:30:55.869110548 +0000 UTC m=+3.969923258 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs") pod "network-metrics-daemon-c4xxk" (UID: "cd501395-aaf2-4005-95bd-f202a13d61b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:30:54.871738 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:54.871698 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:30:55.070917 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:55.070873 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjpw\" (UniqueName: \"kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw\") pod \"network-check-target-m6j5f\" (UID: \"b313af30-1af8-4dac-a6f6-fa42450ea3df\") " pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:30:55.071110 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:55.071071 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:30:55.071110 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:55.071098 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:30:55.071110 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:55.071111 2580 projected.go:196] Error preparing data for projected volume kube-api-access-2wjpw for pod openshift-network-diagnostics/network-check-target-m6j5f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:30:55.071266 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:55.071180 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw podName:b313af30-1af8-4dac-a6f6-fa42450ea3df nodeName:}" failed. No retries permitted until 2026-09-16 13:30:56.071159071 +0000 UTC m=+4.171971784 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2wjpw" (UniqueName: "kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw") pod "network-check-target-m6j5f" (UID: "b313af30-1af8-4dac-a6f6-fa42450ea3df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:30:55.295418 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:55.295255 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 13:25:54 +0000 UTC" deadline="2028-06-03 10:30:13.656590562 +0000 UTC" Sep 16 13:30:55.295418 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:55.295368 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15020h59m18.361227001s" Sep 16 13:30:55.310212 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:55.310182 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55908b4_7d5e_4a7d_8e5b_b84cbac4d4a2.slice/crio-316d9fb415b1abe90ee74ffd5f6d38046a3f353cb657bfa89678ee23c343d037 WatchSource:0}: Error finding container 316d9fb415b1abe90ee74ffd5f6d38046a3f353cb657bfa89678ee23c343d037: Status 404 returned error can't find the container with id 316d9fb415b1abe90ee74ffd5f6d38046a3f353cb657bfa89678ee23c343d037 Sep 16 13:30:55.316219 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:55.315875 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab189943_4b62_4e77_87bd_75934dcb19cb.slice/crio-a484fd8504a6a3cdf55313e7073bfc05de2bc836d0ede261141b3f6cb4097455 WatchSource:0}: Error finding container a484fd8504a6a3cdf55313e7073bfc05de2bc836d0ede261141b3f6cb4097455: Status 404 returned error can't find the container with id a484fd8504a6a3cdf55313e7073bfc05de2bc836d0ede261141b3f6cb4097455 Sep 16 13:30:55.317412 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:55.317312 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3b171d_e589_4e4a_a6f4_e87bc11c6207.slice/crio-7bc198884a1f82706a00b5ab61660e2869364230ad0b3f338ccc19d2da4e89d0 WatchSource:0}: Error finding container 7bc198884a1f82706a00b5ab61660e2869364230ad0b3f338ccc19d2da4e89d0: Status 404 returned error can't find the container with id 7bc198884a1f82706a00b5ab61660e2869364230ad0b3f338ccc19d2da4e89d0 Sep 16 13:30:55.318519 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:55.318478 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3fb5c28_c4c4_4ca2_8fca_340fc4fa5ec5.slice/crio-707c7a93c82eec6492fcbcd0d10571ea1ca3b11cca5f6dfccaf817e57ff07a9d WatchSource:0}: Error finding container 707c7a93c82eec6492fcbcd0d10571ea1ca3b11cca5f6dfccaf817e57ff07a9d: Status 404 returned error can't find the container with id 707c7a93c82eec6492fcbcd0d10571ea1ca3b11cca5f6dfccaf817e57ff07a9d Sep 16 13:30:55.319330 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:55.319238 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5210d309_d43d_4468_861b_7d2a3ac8b4d5.slice/crio-2b1a2491f7fa2a091fb51065b091507b9677d25ca408d7d3676cd3f2c8f06159 WatchSource:0}: Error finding container 2b1a2491f7fa2a091fb51065b091507b9677d25ca408d7d3676cd3f2c8f06159: Status 404 returned error can't find the container with id 2b1a2491f7fa2a091fb51065b091507b9677d25ca408d7d3676cd3f2c8f06159 Sep 16 13:30:55.322670 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:55.320852 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1c10dc_eefa_4040_91e6_c52ce36a7ac5.slice/crio-d29d35074568171f327a5b85e9771bff0d63c0710dac092ec0c5575594fba153 WatchSource:0}: Error finding container d29d35074568171f327a5b85e9771bff0d63c0710dac092ec0c5575594fba153: Status 404 returned error can't find the container with id d29d35074568171f327a5b85e9771bff0d63c0710dac092ec0c5575594fba153 Sep 16 13:30:55.322965 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:30:55.322883 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b34db9_0ba0_4a12_8366_e9bca293d05c.slice/crio-0c00538535776554238849dc2c83545752efc42e6130b67e55b82fecc9be3557 WatchSource:0}: Error finding container 0c00538535776554238849dc2c83545752efc42e6130b67e55b82fecc9be3557: Status 404 returned error can't find the container with id 0c00538535776554238849dc2c83545752efc42e6130b67e55b82fecc9be3557 Sep 16 13:30:55.361063 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:55.361035 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4wwcp" event={"ID":"4c53ff24-05da-4f65-8508-84dda5a72a7d","Type":"ContainerStarted","Data":"893be1a72480a6dfbfc1392537bec795ee6063bc9f7d3b42f673cd200a163b20"} Sep 16 13:30:55.362909 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:55.362882 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mslhp" event={"ID":"3b1c10dc-eefa-4040-91e6-c52ce36a7ac5","Type":"ContainerStarted","Data":"d29d35074568171f327a5b85e9771bff0d63c0710dac092ec0c5575594fba153"} Sep 16 13:30:55.365141 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:55.365106 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" event={"ID":"5210d309-d43d-4468-861b-7d2a3ac8b4d5","Type":"ContainerStarted","Data":"2b1a2491f7fa2a091fb51065b091507b9677d25ca408d7d3676cd3f2c8f06159"} Sep 16 13:30:55.380235 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:55.380202 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" event={"ID":"7b3b171d-e589-4e4a-a6f4-e87bc11c6207","Type":"ContainerStarted","Data":"7bc198884a1f82706a00b5ab61660e2869364230ad0b3f338ccc19d2da4e89d0"} Sep 16 13:30:55.382938 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:55.382907 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" event={"ID":"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5","Type":"ContainerStarted","Data":"707c7a93c82eec6492fcbcd0d10571ea1ca3b11cca5f6dfccaf817e57ff07a9d"} Sep 16 13:30:55.387679 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:55.387641 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mf77j" event={"ID":"ab189943-4b62-4e77-87bd-75934dcb19cb","Type":"ContainerStarted","Data":"a484fd8504a6a3cdf55313e7073bfc05de2bc836d0ede261141b3f6cb4097455"} Sep 16 13:30:55.389930 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:55.389867 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-97dc7" event={"ID":"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2","Type":"ContainerStarted","Data":"316d9fb415b1abe90ee74ffd5f6d38046a3f353cb657bfa89678ee23c343d037"} Sep 16 13:30:55.391083 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:55.391058 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wml9m" event={"ID":"70b7167e-0042-4b50-905c-2cb444c08bc3","Type":"ContainerStarted","Data":"909bb22ba0d7e9add64672c809ebc11da1d464a26ca2f5126b043e68cd822962"} Sep 16 13:30:55.392443 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:55.392417 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wdrqt" event={"ID":"50b34db9-0ba0-4a12-8366-e9bca293d05c","Type":"ContainerStarted","Data":"0c00538535776554238849dc2c83545752efc42e6130b67e55b82fecc9be3557"} Sep 16 13:30:55.877327 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:55.877236 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs\") pod \"network-metrics-daemon-c4xxk\" (UID: \"cd501395-aaf2-4005-95bd-f202a13d61b5\") " pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:30:55.877327 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:55.877312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret\") pod \"global-pull-secret-syncer-cmvcd\" (UID: \"5f469d5f-6b1f-4f0e-910d-bff21be9a372\") " pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:30:55.877570 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:55.877450 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:30:55.877570 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:55.877520 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret podName:5f469d5f-6b1f-4f0e-910d-bff21be9a372 nodeName:}" failed. No retries permitted until 2026-09-16 13:30:57.877499849 +0000 UTC m=+5.978312558 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret") pod "global-pull-secret-syncer-cmvcd" (UID: "5f469d5f-6b1f-4f0e-910d-bff21be9a372") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:30:55.877958 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:55.877937 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:30:55.878077 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:55.877995 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs podName:cd501395-aaf2-4005-95bd-f202a13d61b5 nodeName:}" failed. No retries permitted until 2026-09-16 13:30:57.877978998 +0000 UTC m=+5.978791715 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs") pod "network-metrics-daemon-c4xxk" (UID: "cd501395-aaf2-4005-95bd-f202a13d61b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:30:56.079404 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:56.079349 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjpw\" (UniqueName: \"kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw\") pod \"network-check-target-m6j5f\" (UID: \"b313af30-1af8-4dac-a6f6-fa42450ea3df\") " pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:30:56.079633 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:56.079594 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:30:56.079727 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:56.079647 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:30:56.079727 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:56.079679 2580 projected.go:196] Error preparing data for projected volume kube-api-access-2wjpw for pod openshift-network-diagnostics/network-check-target-m6j5f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:30:56.079853 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:56.079753 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw podName:b313af30-1af8-4dac-a6f6-fa42450ea3df nodeName:}" failed. No retries permitted until 2026-09-16 13:30:58.079731088 +0000 UTC m=+6.180543809 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2wjpw" (UniqueName: "kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw") pod "network-check-target-m6j5f" (UID: "b313af30-1af8-4dac-a6f6-fa42450ea3df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:30:56.296022 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:56.295853 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 13:25:54 +0000 UTC" deadline="2028-04-01 16:14:15.366884981 +0000 UTC" Sep 16 13:30:56.296022 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:56.295919 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13514h43m19.070970626s" Sep 16 13:30:56.352642 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:56.351420 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:30:56.352642 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:56.351886 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmvcd" podUID="5f469d5f-6b1f-4f0e-910d-bff21be9a372" Sep 16 13:30:56.352642 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:56.352305 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:30:56.352642 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:56.352415 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6j5f" podUID="b313af30-1af8-4dac-a6f6-fa42450ea3df" Sep 16 13:30:56.352642 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:56.352498 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:30:56.352642 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:56.352585 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4xxk" podUID="cd501395-aaf2-4005-95bd-f202a13d61b5" Sep 16 13:30:56.412449 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:56.412414 2580 generic.go:356] "Generic (PLEG): container finished" podID="041a12f64202249206ea5b6ffae99644" containerID="e7bd5846c5f7b45e17852fbdaa09a0451e815e3192eef293bfe3e1bd50025163" exitCode=0 Sep 16 13:30:56.412654 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:56.412512 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" event={"ID":"041a12f64202249206ea5b6ffae99644","Type":"ContainerDied","Data":"e7bd5846c5f7b45e17852fbdaa09a0451e815e3192eef293bfe3e1bd50025163"} Sep 16 13:30:56.439356 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:56.438780 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-9.ec2.internal" event={"ID":"4fc4f536cfbc959096a0f31f1c1c0873","Type":"ContainerStarted","Data":"6909554e282c2b15e6fc9b5900bd183d4d3da027bee3cbe139b604650c8050e8"} Sep 16 13:30:57.468594 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:57.468143 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" event={"ID":"041a12f64202249206ea5b6ffae99644","Type":"ContainerStarted","Data":"a7448239a94bfdabb2d8e0ea2129f3c126e5594a8924585a1a7d1e6f056e3952"} Sep 16 13:30:57.491013 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:57.490041 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-9.ec2.internal" podStartSLOduration=3.490017507 podStartE2EDuration="3.490017507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:30:56.457095153 +0000 UTC m=+4.557907875" watchObservedRunningTime="2026-09-16 13:30:57.490017507 +0000 UTC m=+5.590830227" Sep 16 13:30:57.895648 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:57.894739 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs\") pod \"network-metrics-daemon-c4xxk\" (UID: \"cd501395-aaf2-4005-95bd-f202a13d61b5\") " pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:30:57.895648 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:57.894822 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret\") pod \"global-pull-secret-syncer-cmvcd\" (UID: \"5f469d5f-6b1f-4f0e-910d-bff21be9a372\") " pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:30:57.895648 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:57.895013 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:30:57.895648 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:57.895091 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret podName:5f469d5f-6b1f-4f0e-910d-bff21be9a372 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:01.89506806 +0000 UTC m=+9.995880781 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret") pod "global-pull-secret-syncer-cmvcd" (UID: "5f469d5f-6b1f-4f0e-910d-bff21be9a372") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:30:57.895648 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:57.895505 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:30:57.895648 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:57.895560 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs podName:cd501395-aaf2-4005-95bd-f202a13d61b5 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:01.895544083 +0000 UTC m=+9.996356798 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs") pod "network-metrics-daemon-c4xxk" (UID: "cd501395-aaf2-4005-95bd-f202a13d61b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:30:58.097066 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:58.097022 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjpw\" (UniqueName: \"kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw\") pod \"network-check-target-m6j5f\" (UID: \"b313af30-1af8-4dac-a6f6-fa42450ea3df\") " pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:30:58.097259 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:58.097244 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:30:58.097336 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:58.097268 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:30:58.097336 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:58.097281 2580 projected.go:196] Error preparing data for projected volume kube-api-access-2wjpw for pod openshift-network-diagnostics/network-check-target-m6j5f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:30:58.097445 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:58.097341 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw podName:b313af30-1af8-4dac-a6f6-fa42450ea3df nodeName:}" failed. No retries permitted until 2026-09-16 13:31:02.097322938 +0000 UTC m=+10.198135638 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2wjpw" (UniqueName: "kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw") pod "network-check-target-m6j5f" (UID: "b313af30-1af8-4dac-a6f6-fa42450ea3df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:30:58.349920 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:58.349836 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:30:58.350096 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:58.349988 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmvcd" podUID="5f469d5f-6b1f-4f0e-910d-bff21be9a372" Sep 16 13:30:58.350424 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:58.350401 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:30:58.350550 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:58.350527 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6j5f" podUID="b313af30-1af8-4dac-a6f6-fa42450ea3df" Sep 16 13:30:58.353482 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:30:58.353193 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:30:58.353482 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:30:58.353309 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4xxk" podUID="cd501395-aaf2-4005-95bd-f202a13d61b5" Sep 16 13:31:00.351154 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:00.350699 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:31:00.351154 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:00.350699 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:31:00.351154 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:00.350822 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmvcd" podUID="5f469d5f-6b1f-4f0e-910d-bff21be9a372" Sep 16 13:31:00.351154 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:00.350847 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:31:00.351154 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:00.350936 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6j5f" podUID="b313af30-1af8-4dac-a6f6-fa42450ea3df" Sep 16 13:31:00.351154 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:00.351023 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4xxk" podUID="cd501395-aaf2-4005-95bd-f202a13d61b5" Sep 16 13:31:01.930087 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:01.930032 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret\") pod \"global-pull-secret-syncer-cmvcd\" (UID: \"5f469d5f-6b1f-4f0e-910d-bff21be9a372\") " pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:31:01.930565 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:01.930114 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs\") pod \"network-metrics-daemon-c4xxk\" (UID: \"cd501395-aaf2-4005-95bd-f202a13d61b5\") " pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:31:01.930565 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:01.930264 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:31:01.930565 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:01.930325 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs podName:cd501395-aaf2-4005-95bd-f202a13d61b5 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:09.930306338 +0000 UTC m=+18.031119045 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs") pod "network-metrics-daemon-c4xxk" (UID: "cd501395-aaf2-4005-95bd-f202a13d61b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:31:01.930762 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:01.930731 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:31:01.930871 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:01.930784 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret podName:5f469d5f-6b1f-4f0e-910d-bff21be9a372 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:09.930768175 +0000 UTC m=+18.031580893 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret") pod "global-pull-secret-syncer-cmvcd" (UID: "5f469d5f-6b1f-4f0e-910d-bff21be9a372") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:31:02.132467 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:02.131935 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjpw\" (UniqueName: \"kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw\") pod \"network-check-target-m6j5f\" (UID: \"b313af30-1af8-4dac-a6f6-fa42450ea3df\") " pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:31:02.132467 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:02.132078 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:31:02.132467 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:02.132099 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:31:02.132467 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:02.132112 2580 projected.go:196] Error preparing data for projected volume kube-api-access-2wjpw for pod openshift-network-diagnostics/network-check-target-m6j5f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:31:02.132467 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:02.132168 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw podName:b313af30-1af8-4dac-a6f6-fa42450ea3df nodeName:}" failed. No retries permitted until 2026-09-16 13:31:10.132151334 +0000 UTC m=+18.232964030 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2wjpw" (UniqueName: "kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw") pod "network-check-target-m6j5f" (UID: "b313af30-1af8-4dac-a6f6-fa42450ea3df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:31:02.352152 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:02.351397 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:31:02.352152 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:02.351530 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmvcd" podUID="5f469d5f-6b1f-4f0e-910d-bff21be9a372" Sep 16 13:31:02.352152 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:02.351579 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:31:02.352152 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:02.351691 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6j5f" podUID="b313af30-1af8-4dac-a6f6-fa42450ea3df" Sep 16 13:31:02.352152 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:02.351733 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:31:02.352152 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:02.351787 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4xxk" podUID="cd501395-aaf2-4005-95bd-f202a13d61b5" Sep 16 13:31:04.350087 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:04.349004 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:31:04.350087 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:04.349148 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4xxk" podUID="cd501395-aaf2-4005-95bd-f202a13d61b5" Sep 16 13:31:04.350087 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:04.349842 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:31:04.350087 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:04.349850 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:31:04.350087 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:04.349940 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmvcd" podUID="5f469d5f-6b1f-4f0e-910d-bff21be9a372" Sep 16 13:31:04.350087 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:04.350018 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6j5f" podUID="b313af30-1af8-4dac-a6f6-fa42450ea3df" Sep 16 13:31:06.349506 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:06.349467 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:31:06.349948 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:06.349471 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:31:06.349948 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:06.349600 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmvcd" podUID="5f469d5f-6b1f-4f0e-910d-bff21be9a372" Sep 16 13:31:06.349948 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:06.349467 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:31:06.349948 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:06.349672 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4xxk" podUID="cd501395-aaf2-4005-95bd-f202a13d61b5" Sep 16 13:31:06.349948 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:06.349734 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6j5f" podUID="b313af30-1af8-4dac-a6f6-fa42450ea3df" Sep 16 13:31:08.349376 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:08.349341 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:31:08.349775 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:08.349343 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:31:08.349775 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:08.349479 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:31:08.349775 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:08.349472 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6j5f" podUID="b313af30-1af8-4dac-a6f6-fa42450ea3df" Sep 16 13:31:08.349775 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:08.349586 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4xxk" podUID="cd501395-aaf2-4005-95bd-f202a13d61b5" Sep 16 13:31:08.349775 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:08.349669 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmvcd" podUID="5f469d5f-6b1f-4f0e-910d-bff21be9a372" Sep 16 13:31:09.988322 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:09.988283 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret\") pod \"global-pull-secret-syncer-cmvcd\" (UID: \"5f469d5f-6b1f-4f0e-910d-bff21be9a372\") " pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:31:09.988868 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:09.988349 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs\") pod \"network-metrics-daemon-c4xxk\" (UID: \"cd501395-aaf2-4005-95bd-f202a13d61b5\") " pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:31:09.988868 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:09.988460 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:31:09.988868 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:09.988526 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret podName:5f469d5f-6b1f-4f0e-910d-bff21be9a372 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:25.988506889 +0000 UTC m=+34.089319595 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret") pod "global-pull-secret-syncer-cmvcd" (UID: "5f469d5f-6b1f-4f0e-910d-bff21be9a372") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:31:09.988868 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:09.988460 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:31:09.988868 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:09.988584 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs podName:cd501395-aaf2-4005-95bd-f202a13d61b5 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:25.988571879 +0000 UTC m=+34.089384573 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs") pod "network-metrics-daemon-c4xxk" (UID: "cd501395-aaf2-4005-95bd-f202a13d61b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:31:10.190725 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:10.190687 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjpw\" (UniqueName: \"kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw\") pod \"network-check-target-m6j5f\" (UID: \"b313af30-1af8-4dac-a6f6-fa42450ea3df\") " pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:31:10.190885 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:10.190860 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:31:10.190885 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:10.190882 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:31:10.190958 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:10.190892 2580 projected.go:196] Error preparing data for projected volume kube-api-access-2wjpw for pod openshift-network-diagnostics/network-check-target-m6j5f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:31:10.190958 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:10.190949 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw podName:b313af30-1af8-4dac-a6f6-fa42450ea3df nodeName:}" failed. No retries permitted until 2026-09-16 13:31:26.190935546 +0000 UTC m=+34.291748253 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2wjpw" (UniqueName: "kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw") pod "network-check-target-m6j5f" (UID: "b313af30-1af8-4dac-a6f6-fa42450ea3df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:31:10.349122 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:10.349030 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:31:10.349122 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:10.349081 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:31:10.349122 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:10.349085 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:31:10.349350 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:10.349170 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmvcd" podUID="5f469d5f-6b1f-4f0e-910d-bff21be9a372" Sep 16 13:31:10.349350 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:10.349273 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6j5f" podUID="b313af30-1af8-4dac-a6f6-fa42450ea3df" Sep 16 13:31:10.349427 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:10.349357 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4xxk" podUID="cd501395-aaf2-4005-95bd-f202a13d61b5" Sep 16 13:31:12.350303 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:12.350268 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:31:12.350814 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:12.350385 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:31:12.350814 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:12.350508 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmvcd" podUID="5f469d5f-6b1f-4f0e-910d-bff21be9a372" Sep 16 13:31:12.350814 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:12.350388 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4xxk" podUID="cd501395-aaf2-4005-95bd-f202a13d61b5" Sep 16 13:31:12.350814 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:12.350564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:31:12.350814 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:12.350633 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6j5f" podUID="b313af30-1af8-4dac-a6f6-fa42450ea3df" Sep 16 13:31:14.349964 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.349529 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:31:14.350645 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.349538 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:31:14.350645 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:14.350026 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmvcd" podUID="5f469d5f-6b1f-4f0e-910d-bff21be9a372" Sep 16 13:31:14.350645 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:14.350096 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6j5f" podUID="b313af30-1af8-4dac-a6f6-fa42450ea3df" Sep 16 13:31:14.350645 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.349595 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:31:14.350645 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:14.350208 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4xxk" podUID="cd501395-aaf2-4005-95bd-f202a13d61b5" Sep 16 13:31:14.506237 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.506202 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" event={"ID":"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5","Type":"ContainerStarted","Data":"0491a2eb7075e9f0e86b53b7fd7798e525c74ec1f82e7682f19de9f0696e9910"} Sep 16 13:31:14.506383 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.506299 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" event={"ID":"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5","Type":"ContainerStarted","Data":"496f1bc2f87b1165ee8a5862dac0ef98e7c07bd43bd288211723308a9a26f022"} Sep 16 13:31:14.506383 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.506353 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" event={"ID":"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5","Type":"ContainerStarted","Data":"27884326a908dcd383807f1e20d9c9e175667c21e8b032ca45e5ead895cc1b4c"} Sep 16 13:31:14.506467 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.506399 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" event={"ID":"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5","Type":"ContainerStarted","Data":"d976f1e4a80d9aabcb48f74b9cc46bcdb2756dc738a63a91e1db4de23b7d5d1e"} Sep 16 13:31:14.506467 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.506447 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" event={"ID":"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5","Type":"ContainerStarted","Data":"2c24eebd44b106856b1a66705717dba4aa2d56aae55b025e556467dbd5e97ec3"} Sep 16 13:31:14.506561 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.506481 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" event={"ID":"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5","Type":"ContainerStarted","Data":"cb95a66d0f088423969c03d1ad2961bb68b407183cf8b16e33ddb64c81426ac9"} Sep 16 13:31:14.507848 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.507819 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mf77j" event={"ID":"ab189943-4b62-4e77-87bd-75934dcb19cb","Type":"ContainerStarted","Data":"6afdd8ac81fc05a96af7a308c2fcdf2444ccae5bce036c255d1707e5a4f5bb1d"} Sep 16 13:31:14.509201 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.509176 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-97dc7" event={"ID":"d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2","Type":"ContainerStarted","Data":"16631a32d3e0c5cead94577b784cc12a89b7ea32cc22e298a847a80f649cb468"} Sep 16 13:31:14.510545 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.510522 2580 generic.go:356] "Generic (PLEG): container finished" podID="70b7167e-0042-4b50-905c-2cb444c08bc3" containerID="e7fb4e33453e256a3724845c741d03be315eeccd36934dc3c15a2f7b31c70da2" exitCode=0 Sep 16 13:31:14.510656 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.510586 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wml9m" event={"ID":"70b7167e-0042-4b50-905c-2cb444c08bc3","Type":"ContainerDied","Data":"e7fb4e33453e256a3724845c741d03be315eeccd36934dc3c15a2f7b31c70da2"} Sep 16 13:31:14.512846 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.512826 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4wwcp" event={"ID":"4c53ff24-05da-4f65-8508-84dda5a72a7d","Type":"ContainerStarted","Data":"9b4dbb3ff1e4ce69198dc5592985e3a05a8a8fdbedf2f00249649ad2e7db9f60"} Sep 16 13:31:14.514393 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.514371 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mslhp" event={"ID":"3b1c10dc-eefa-4040-91e6-c52ce36a7ac5","Type":"ContainerStarted","Data":"b3f749d6c4d333cb35a30d90b890b52c4797308a99aac60d3ae4b6d7191d0bd8"} Sep 16 13:31:14.515824 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.515806 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" event={"ID":"5210d309-d43d-4468-861b-7d2a3ac8b4d5","Type":"ContainerStarted","Data":"49928fc4997463183d14e74462488cda899d15ae7f2b9f9fb7d05d563eefb40e"} Sep 16 13:31:14.517205 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.517185 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" event={"ID":"7b3b171d-e589-4e4a-a6f4-e87bc11c6207","Type":"ContainerStarted","Data":"d0f2c52d43658a61e0dcd75735ec716103a77ff8b04c60fcabfed97d6b08a550"} Sep 16 13:31:14.526172 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.526136 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mf77j" podStartSLOduration=4.3445802350000005 podStartE2EDuration="22.526124885s" totalImagesPullingTime="18.18154465s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:14.525422141 +0000 UTC m=+22.626234857" watchObservedRunningTime="2026-09-16 13:31:14.526124885 +0000 UTC m=+22.626937601" Sep 16 13:31:14.526327 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.526309 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-9.ec2.internal" podStartSLOduration=20.526303157 podStartE2EDuration="20.526303157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:30:57.490994361 +0000 UTC m=+5.591807082" watchObservedRunningTime="2026-09-16 13:31:14.526303157 +0000 UTC m=+22.627115872" Sep 16 13:31:14.543186 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.543123 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mslhp" podStartSLOduration=4.398952046 podStartE2EDuration="22.543107163s" totalImagesPullingTime="18.144155117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:14.542822625 +0000 UTC m=+22.643635337" watchObservedRunningTime="2026-09-16 13:31:14.543107163 +0000 UTC m=+22.643919877" Sep 16 13:31:14.559922 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.559852 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4wwcp" podStartSLOduration=4.396621785 podStartE2EDuration="22.559829504s" totalImagesPullingTime="18.163207719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:14.559370729 +0000 UTC m=+22.660183445" watchObservedRunningTime="2026-09-16 13:31:14.559829504 +0000 UTC m=+22.660642222" Sep 16 13:31:14.588428 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.588370 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-97dc7" podStartSLOduration=4.340517019 podStartE2EDuration="22.588356149s" totalImagesPullingTime="18.24783913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:14.587993871 +0000 UTC m=+22.688806590" watchObservedRunningTime="2026-09-16 13:31:14.588356149 +0000 UTC m=+22.689168864" Sep 16 13:31:14.607353 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.607256 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xdbpd" podStartSLOduration=4.419410854 podStartE2EDuration="22.607242399s" totalImagesPullingTime="18.187831545s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:14.607014396 +0000 UTC m=+22.707827110" watchObservedRunningTime="2026-09-16 13:31:14.607242399 +0000 UTC m=+22.708055124" Sep 16 13:31:14.740545 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.740515 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:14.819210 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.819179 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:14.920728 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:14.920661 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:15.076800 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:15.076770 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 13:31:15.248537 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:15.248501 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:15.322982 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:15.322832 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T13:31:15.076791688Z","UUID":"54e64c28-6945-4004-bbd7-06248a19e05e","Handler":null,"Name":"","Endpoint":""} Sep 16 13:31:15.324407 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:15.324383 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 13:31:15.324556 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:15.324434 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 13:31:15.350676 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:15.350651 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:15.449942 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:15.449912 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:15.522319 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:15.522233 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" event={"ID":"5210d309-d43d-4468-861b-7d2a3ac8b4d5","Type":"ContainerStarted","Data":"f3296f5c19199858c84c57661af40ed1e73a8256a7783d8d6589711811b0b80d"} Sep 16 13:31:15.524110 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:15.524078 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wdrqt" event={"ID":"50b34db9-0ba0-4a12-8366-e9bca293d05c","Type":"ContainerStarted","Data":"0a1d892aa66a0f8eaf1d53cdeed15eec855b4b20f37c4df951415bd9cd2a8572"} Sep 16 13:31:15.541429 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:15.541371 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wdrqt" podStartSLOduration=5.374126368 podStartE2EDuration="23.541352859s" totalImagesPullingTime="18.167226491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:15.540833857 +0000 UTC m=+23.641646572" watchObservedRunningTime="2026-09-16 13:31:15.541352859 +0000 UTC m=+23.642165575" Sep 16 13:31:15.877897 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:15.877796 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:16.132467 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:16.132380 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4wwcp" Sep 16 13:31:16.275107 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:16.275081 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:16.340927 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:16.340901 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:16.349072 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:16.349043 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:31:16.349244 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:16.349045 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:31:16.349244 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:16.349175 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:31:16.349244 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:16.349186 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmvcd" podUID="5f469d5f-6b1f-4f0e-910d-bff21be9a372" Sep 16 13:31:16.349413 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:16.349306 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4xxk" podUID="cd501395-aaf2-4005-95bd-f202a13d61b5" Sep 16 13:31:16.349413 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:16.349376 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6j5f" podUID="b313af30-1af8-4dac-a6f6-fa42450ea3df" Sep 16 13:31:16.530642 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:16.530595 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" event={"ID":"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5","Type":"ContainerStarted","Data":"d6b98fe34cc5e3dcb56fbe10bc41473f72398ea2d4d32fd85151d57f4ca3730e"} Sep 16 13:31:17.083174 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:17.082992 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:17.268658 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:17.268618 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4wwcp" Sep 16 13:31:17.269658 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:17.269635 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4wwcp" Sep 16 13:31:17.470289 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:17.470260 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:17.536459 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:17.536423 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" event={"ID":"5210d309-d43d-4468-861b-7d2a3ac8b4d5","Type":"ContainerStarted","Data":"cec754cb0835fba23201ac9619434119fd6259644b9fbeba1c9ec67ad4166da6"} Sep 16 13:31:17.536981 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:17.536818 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:17.537422 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:17.537402 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4wwcp" Sep 16 13:31:17.557327 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:17.557274 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5jv6" podStartSLOduration=4.823390959 podStartE2EDuration="25.557256974s" totalImagesPullingTime="20.733866015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:30:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:17.55612049 +0000 UTC m=+25.656933205" watchObservedRunningTime="2026-09-16 13:31:17.557256974 +0000 UTC m=+25.658069688" Sep 16 13:31:17.635320 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:17.635281 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:18.283165 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:18.283133 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:18.344708 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:18.344678 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:18.349081 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:18.349052 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:31:18.349205 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:18.349166 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmvcd" podUID="5f469d5f-6b1f-4f0e-910d-bff21be9a372" Sep 16 13:31:18.349561 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:18.349544 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:31:18.349664 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:18.349642 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6j5f" podUID="b313af30-1af8-4dac-a6f6-fa42450ea3df" Sep 16 13:31:18.349996 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:18.349979 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:31:18.350116 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:18.350096 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4xxk" podUID="cd501395-aaf2-4005-95bd-f202a13d61b5" Sep 16 13:31:18.678145 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:18.678111 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:18.733507 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:18.733472 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:20.215680 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:20.215446 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:20.349245 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:20.349209 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:31:20.349367 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:20.349222 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:31:20.349418 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:20.349367 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:31:20.349418 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:20.349369 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4xxk" podUID="cd501395-aaf2-4005-95bd-f202a13d61b5" Sep 16 13:31:20.349484 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:20.349448 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmvcd" podUID="5f469d5f-6b1f-4f0e-910d-bff21be9a372" Sep 16 13:31:20.349535 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:20.349517 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6j5f" podUID="b313af30-1af8-4dac-a6f6-fa42450ea3df" Sep 16 13:31:20.545917 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:20.545838 2580 generic.go:356] "Generic (PLEG): container finished" podID="70b7167e-0042-4b50-905c-2cb444c08bc3" containerID="03f2ca7bd44856869b05ec12268f860544399ebbf3541ed55773eb21d76b03fc" exitCode=0 Sep 16 13:31:20.546054 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:20.545917 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wml9m" event={"ID":"70b7167e-0042-4b50-905c-2cb444c08bc3","Type":"ContainerDied","Data":"03f2ca7bd44856869b05ec12268f860544399ebbf3541ed55773eb21d76b03fc"} Sep 16 13:31:20.549775 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:20.549746 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" event={"ID":"f3fb5c28-c4c4-4ca2-8fca-340fc4fa5ec5","Type":"ContainerStarted","Data":"fe09190a5a0cd53337fa0fb8c5388a1f0168f435172c516e5c89b9acb384f86a"} Sep 16 13:31:20.550061 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:20.550005 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:31:20.550113 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:20.550062 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:31:20.565947 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:20.565924 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:31:20.605416 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:20.605365 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" podStartSLOduration=10.382140869 podStartE2EDuration="28.605348856s" totalImagesPullingTime="18.223207987s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:20.603658383 +0000 UTC m=+28.704471097" watchObservedRunningTime="2026-09-16 13:31:20.605348856 +0000 UTC m=+28.706161571" Sep 16 13:31:21.552626 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:21.552577 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:31:21.568379 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:21.568346 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:31:21.589085 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:21.589058 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:21.763061 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:21.763029 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:21.864209 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:21.864169 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:22.349940 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:22.349910 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:31:22.350084 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:22.349998 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:31:22.350084 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:22.350042 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:31:22.350084 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:22.350048 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmvcd" podUID="5f469d5f-6b1f-4f0e-910d-bff21be9a372" Sep 16 13:31:22.350234 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:22.350114 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6j5f" podUID="b313af30-1af8-4dac-a6f6-fa42450ea3df" Sep 16 13:31:22.350234 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:22.350213 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4xxk" podUID="cd501395-aaf2-4005-95bd-f202a13d61b5" Sep 16 13:31:22.424931 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:22.424711 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m6j5f"] Sep 16 13:31:22.425544 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:22.425518 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cmvcd"] Sep 16 13:31:22.439540 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:22.439511 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c4xxk"] Sep 16 13:31:22.557444 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:22.557410 2580 generic.go:356] "Generic (PLEG): container finished" podID="70b7167e-0042-4b50-905c-2cb444c08bc3" containerID="7eeac6d8fe3747f36d8b2f6b537967bd16e892ffce4a4e7874b8120c93a83e46" exitCode=0 Sep 16 13:31:22.557922 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:22.557485 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wml9m" event={"ID":"70b7167e-0042-4b50-905c-2cb444c08bc3","Type":"ContainerDied","Data":"7eeac6d8fe3747f36d8b2f6b537967bd16e892ffce4a4e7874b8120c93a83e46"} Sep 16 13:31:22.557922 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:22.557538 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:31:22.557922 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:22.557651 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6j5f" podUID="b313af30-1af8-4dac-a6f6-fa42450ea3df" Sep 16 13:31:22.557922 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:22.557661 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:31:22.557922 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:22.557666 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:31:22.557922 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:22.557743 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmvcd" podUID="5f469d5f-6b1f-4f0e-910d-bff21be9a372" Sep 16 13:31:22.557922 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:22.557859 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4xxk" podUID="cd501395-aaf2-4005-95bd-f202a13d61b5" Sep 16 13:31:24.349905 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:24.349564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:31:24.349905 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:24.349596 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:31:24.349905 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:24.349563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:31:24.349905 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:24.349718 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmvcd" podUID="5f469d5f-6b1f-4f0e-910d-bff21be9a372" Sep 16 13:31:24.349905 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:24.349796 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4xxk" podUID="cd501395-aaf2-4005-95bd-f202a13d61b5" Sep 16 13:31:24.349905 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:24.349882 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6j5f" podUID="b313af30-1af8-4dac-a6f6-fa42450ea3df" Sep 16 13:31:24.564688 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:24.564652 2580 generic.go:356] "Generic (PLEG): container finished" podID="70b7167e-0042-4b50-905c-2cb444c08bc3" containerID="f0a92d9cf3bf35158c73a5ccf4b9d685baaf85b0f8ecd70dce0615a637fb375b" exitCode=0 Sep 16 13:31:24.564836 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:24.564720 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wml9m" event={"ID":"70b7167e-0042-4b50-905c-2cb444c08bc3","Type":"ContainerDied","Data":"f0a92d9cf3bf35158c73a5ccf4b9d685baaf85b0f8ecd70dce0615a637fb375b"} Sep 16 13:31:25.827784 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:25.827736 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:25.926847 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:25.926810 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:26.002431 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.002385 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs\") pod \"network-metrics-daemon-c4xxk\" (UID: \"cd501395-aaf2-4005-95bd-f202a13d61b5\") " pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:31:26.002649 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.002472 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret\") pod \"global-pull-secret-syncer-cmvcd\" (UID: \"5f469d5f-6b1f-4f0e-910d-bff21be9a372\") " pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:31:26.002649 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:26.002547 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:31:26.002649 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:26.002558 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:31:26.002649 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:26.002640 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret podName:5f469d5f-6b1f-4f0e-910d-bff21be9a372 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:58.002622601 +0000 UTC m=+66.103435310 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret") pod "global-pull-secret-syncer-cmvcd" (UID: "5f469d5f-6b1f-4f0e-910d-bff21be9a372") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:31:26.002843 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:26.002664 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs podName:cd501395-aaf2-4005-95bd-f202a13d61b5 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:58.002652675 +0000 UTC m=+66.103465369 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs") pod "network-metrics-daemon-c4xxk" (UID: "cd501395-aaf2-4005-95bd-f202a13d61b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:31:26.204253 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.204157 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjpw\" (UniqueName: \"kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw\") pod \"network-check-target-m6j5f\" (UID: \"b313af30-1af8-4dac-a6f6-fa42450ea3df\") " pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:31:26.204418 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:26.204343 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:31:26.204418 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:26.204371 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:31:26.204418 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:26.204383 2580 projected.go:196] Error preparing data for projected volume kube-api-access-2wjpw for pod openshift-network-diagnostics/network-check-target-m6j5f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:31:26.204536 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:26.204439 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw podName:b313af30-1af8-4dac-a6f6-fa42450ea3df nodeName:}" failed. No retries permitted until 2026-09-16 13:31:58.20442221 +0000 UTC m=+66.305234931 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2wjpw" (UniqueName: "kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw") pod "network-check-target-m6j5f" (UID: "b313af30-1af8-4dac-a6f6-fa42450ea3df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:31:26.244177 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.244154 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-9.ec2.internal" event="NodeReady" Sep 16 13:31:26.244344 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.244279 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 13:31:26.291041 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.291012 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jb79h"] Sep 16 13:31:26.305024 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.304995 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fr876"] Sep 16 13:31:26.305163 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.305129 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jb79h" Sep 16 13:31:26.308407 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.308021 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 13:31:26.308407 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.308339 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 13:31:26.308871 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.308855 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t4m6m\"" Sep 16 13:31:26.315137 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.315115 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jb79h"] Sep 16 13:31:26.315254 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.315153 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fr876"] Sep 16 13:31:26.315254 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.315242 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fr876" Sep 16 13:31:26.318135 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.318114 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 13:31:26.318241 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.318145 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 13:31:26.318457 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.318437 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7jfkh\"" Sep 16 13:31:26.318554 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.318542 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 13:31:26.349288 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.349261 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:31:26.349466 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.349261 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:31:26.349718 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.349262 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:31:26.352999 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.352981 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 13:31:26.353459 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.353441 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4n8lp\"" Sep 16 13:31:26.353459 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.353456 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 13:31:26.353747 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.353441 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 13:31:26.353825 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.353782 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 13:31:26.353888 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.353874 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4nxp8\"" Sep 16 13:31:26.405563 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.405523 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mvdq\" (UniqueName: \"kubernetes.io/projected/bb307e0c-d166-40bb-ae14-2315eb27a09c-kube-api-access-9mvdq\") pod \"dns-default-jb79h\" (UID: \"bb307e0c-d166-40bb-ae14-2315eb27a09c\") " pod="openshift-dns/dns-default-jb79h" Sep 16 13:31:26.405738 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.405622 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb307e0c-d166-40bb-ae14-2315eb27a09c-metrics-tls\") pod \"dns-default-jb79h\" (UID: \"bb307e0c-d166-40bb-ae14-2315eb27a09c\") " pod="openshift-dns/dns-default-jb79h" Sep 16 13:31:26.405738 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.405687 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szzpg\" (UniqueName: \"kubernetes.io/projected/b1b0a620-4bae-4969-916f-3c5019de5654-kube-api-access-szzpg\") pod \"ingress-canary-fr876\" (UID: \"b1b0a620-4bae-4969-916f-3c5019de5654\") " pod="openshift-ingress-canary/ingress-canary-fr876" Sep 16 13:31:26.405868 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.405737 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb307e0c-d166-40bb-ae14-2315eb27a09c-tmp-dir\") pod \"dns-default-jb79h\" (UID: \"bb307e0c-d166-40bb-ae14-2315eb27a09c\") " pod="openshift-dns/dns-default-jb79h" Sep 16 13:31:26.405868 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.405768 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb307e0c-d166-40bb-ae14-2315eb27a09c-config-volume\") pod \"dns-default-jb79h\" (UID: \"bb307e0c-d166-40bb-ae14-2315eb27a09c\") " pod="openshift-dns/dns-default-jb79h" Sep 16 13:31:26.405868 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.405805 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1b0a620-4bae-4969-916f-3c5019de5654-cert\") pod \"ingress-canary-fr876\" (UID: \"b1b0a620-4bae-4969-916f-3c5019de5654\") " pod="openshift-ingress-canary/ingress-canary-fr876" Sep 16 13:31:26.424997 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.424960 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:26.507185 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.507152 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb307e0c-d166-40bb-ae14-2315eb27a09c-config-volume\") pod \"dns-default-jb79h\" (UID: \"bb307e0c-d166-40bb-ae14-2315eb27a09c\") " pod="openshift-dns/dns-default-jb79h" Sep 16 13:31:26.507380 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.507229 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1b0a620-4bae-4969-916f-3c5019de5654-cert\") pod \"ingress-canary-fr876\" (UID: \"b1b0a620-4bae-4969-916f-3c5019de5654\") " pod="openshift-ingress-canary/ingress-canary-fr876" Sep 16 13:31:26.507380 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.507256 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mvdq\" (UniqueName: \"kubernetes.io/projected/bb307e0c-d166-40bb-ae14-2315eb27a09c-kube-api-access-9mvdq\") pod \"dns-default-jb79h\" (UID: \"bb307e0c-d166-40bb-ae14-2315eb27a09c\") " pod="openshift-dns/dns-default-jb79h" Sep 16 13:31:26.507380 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.507290 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb307e0c-d166-40bb-ae14-2315eb27a09c-metrics-tls\") pod \"dns-default-jb79h\" (UID: \"bb307e0c-d166-40bb-ae14-2315eb27a09c\") " pod="openshift-dns/dns-default-jb79h" Sep 16 13:31:26.507380 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.507352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szzpg\" (UniqueName: \"kubernetes.io/projected/b1b0a620-4bae-4969-916f-3c5019de5654-kube-api-access-szzpg\") pod \"ingress-canary-fr876\" (UID: \"b1b0a620-4bae-4969-916f-3c5019de5654\") " pod="openshift-ingress-canary/ingress-canary-fr876" Sep 16 13:31:26.507642 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.507408 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb307e0c-d166-40bb-ae14-2315eb27a09c-tmp-dir\") pod \"dns-default-jb79h\" (UID: \"bb307e0c-d166-40bb-ae14-2315eb27a09c\") " pod="openshift-dns/dns-default-jb79h" Sep 16 13:31:26.507744 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:26.507722 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:31:26.507801 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:26.507793 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb307e0c-d166-40bb-ae14-2315eb27a09c-metrics-tls podName:bb307e0c-d166-40bb-ae14-2315eb27a09c nodeName:}" failed. No retries permitted until 2026-09-16 13:31:27.0077756 +0000 UTC m=+35.108588308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb307e0c-d166-40bb-ae14-2315eb27a09c-metrics-tls") pod "dns-default-jb79h" (UID: "bb307e0c-d166-40bb-ae14-2315eb27a09c") : secret "dns-default-metrics-tls" not found Sep 16 13:31:26.507910 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.507720 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb307e0c-d166-40bb-ae14-2315eb27a09c-tmp-dir\") pod \"dns-default-jb79h\" (UID: \"bb307e0c-d166-40bb-ae14-2315eb27a09c\") " pod="openshift-dns/dns-default-jb79h" Sep 16 13:31:26.507967 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.507918 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb307e0c-d166-40bb-ae14-2315eb27a09c-config-volume\") pod \"dns-default-jb79h\" (UID: \"bb307e0c-d166-40bb-ae14-2315eb27a09c\") " pod="openshift-dns/dns-default-jb79h" Sep 16 13:31:26.512229 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.512186 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1b0a620-4bae-4969-916f-3c5019de5654-cert\") pod \"ingress-canary-fr876\" (UID: \"b1b0a620-4bae-4969-916f-3c5019de5654\") " pod="openshift-ingress-canary/ingress-canary-fr876" Sep 16 13:31:26.516783 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.516759 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mvdq\" (UniqueName: \"kubernetes.io/projected/bb307e0c-d166-40bb-ae14-2315eb27a09c-kube-api-access-9mvdq\") pod \"dns-default-jb79h\" (UID: \"bb307e0c-d166-40bb-ae14-2315eb27a09c\") " pod="openshift-dns/dns-default-jb79h" Sep 16 13:31:26.517025 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.517002 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szzpg\" (UniqueName: \"kubernetes.io/projected/b1b0a620-4bae-4969-916f-3c5019de5654-kube-api-access-szzpg\") pod \"ingress-canary-fr876\" (UID: \"b1b0a620-4bae-4969-916f-3c5019de5654\") " pod="openshift-ingress-canary/ingress-canary-fr876" Sep 16 13:31:26.526144 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.526121 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:26.624872 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.624843 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fr876" Sep 16 13:31:26.964412 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:26.964243 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fr876"] Sep 16 13:31:26.968503 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:31:26.968476 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b0a620_4bae_4969_916f_3c5019de5654.slice/crio-66c5cfcd33391787e6d277f3683f209e280ca2d43a89bf56de38d77349a5f5b8 WatchSource:0}: Error finding container 66c5cfcd33391787e6d277f3683f209e280ca2d43a89bf56de38d77349a5f5b8: Status 404 returned error can't find the container with id 66c5cfcd33391787e6d277f3683f209e280ca2d43a89bf56de38d77349a5f5b8 Sep 16 13:31:27.011657 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:27.011623 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb307e0c-d166-40bb-ae14-2315eb27a09c-metrics-tls\") pod \"dns-default-jb79h\" (UID: \"bb307e0c-d166-40bb-ae14-2315eb27a09c\") " pod="openshift-dns/dns-default-jb79h" Sep 16 13:31:27.011820 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:27.011724 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:31:27.011820 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:27.011789 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb307e0c-d166-40bb-ae14-2315eb27a09c-metrics-tls podName:bb307e0c-d166-40bb-ae14-2315eb27a09c nodeName:}" failed. No retries permitted until 2026-09-16 13:31:28.01177419 +0000 UTC m=+36.112586882 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb307e0c-d166-40bb-ae14-2315eb27a09c-metrics-tls") pod "dns-default-jb79h" (UID: "bb307e0c-d166-40bb-ae14-2315eb27a09c") : secret "dns-default-metrics-tls" not found Sep 16 13:31:27.160808 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:27.160777 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:27.220116 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:27.220015 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:27.572241 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:27.572155 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fr876" event={"ID":"b1b0a620-4bae-4969-916f-3c5019de5654","Type":"ContainerStarted","Data":"66c5cfcd33391787e6d277f3683f209e280ca2d43a89bf56de38d77349a5f5b8"} Sep 16 13:31:27.801844 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:27.801806 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:27.895014 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:27.894937 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:27.996208 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:27.996174 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bwssq"] Sep 16 13:31:28.011570 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.011532 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:28.011848 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.011829 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bwssq"] Sep 16 13:31:28.014343 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.014319 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 13:31:28.014920 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.014736 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7rd5w\"" Sep 16 13:31:28.015015 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.014971 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 13:31:28.015461 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.015431 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 13:31:28.015551 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.015505 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 13:31:28.019465 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.019446 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb307e0c-d166-40bb-ae14-2315eb27a09c-metrics-tls\") pod \"dns-default-jb79h\" (UID: \"bb307e0c-d166-40bb-ae14-2315eb27a09c\") " pod="openshift-dns/dns-default-jb79h" Sep 16 13:31:28.019632 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:28.019591 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:31:28.019743 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:28.019686 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb307e0c-d166-40bb-ae14-2315eb27a09c-metrics-tls podName:bb307e0c-d166-40bb-ae14-2315eb27a09c nodeName:}" failed. No retries permitted until 2026-09-16 13:31:30.019668693 +0000 UTC m=+38.120481403 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb307e0c-d166-40bb-ae14-2315eb27a09c-metrics-tls") pod "dns-default-jb79h" (UID: "bb307e0c-d166-40bb-ae14-2315eb27a09c") : secret "dns-default-metrics-tls" not found Sep 16 13:31:28.020930 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.020752 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:28.099703 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.099668 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:28.120846 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.120808 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97b3faa7-090c-4ad2-975d-2722eaaa0d51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:28.121034 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.120891 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97b3faa7-090c-4ad2-975d-2722eaaa0d51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:28.121034 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.120928 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97b3faa7-090c-4ad2-975d-2722eaaa0d51-crio-socket\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:28.121034 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.120975 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9l99\" (UniqueName: \"kubernetes.io/projected/97b3faa7-090c-4ad2-975d-2722eaaa0d51-kube-api-access-w9l99\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:28.121034 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.121003 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/97b3faa7-090c-4ad2-975d-2722eaaa0d51-data-volume\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:28.221970 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.221922 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9l99\" (UniqueName: \"kubernetes.io/projected/97b3faa7-090c-4ad2-975d-2722eaaa0d51-kube-api-access-w9l99\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:28.222191 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.221979 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/97b3faa7-090c-4ad2-975d-2722eaaa0d51-data-volume\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:28.222191 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.222052 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97b3faa7-090c-4ad2-975d-2722eaaa0d51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:28.222191 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.222104 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97b3faa7-090c-4ad2-975d-2722eaaa0d51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:28.222191 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.222172 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97b3faa7-090c-4ad2-975d-2722eaaa0d51-crio-socket\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:28.222387 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:28.222209 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:31:28.222387 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:28.222225 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 13:31:28.222387 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:28.222280 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97b3faa7-090c-4ad2-975d-2722eaaa0d51-insights-runtime-extractor-tls podName:97b3faa7-090c-4ad2-975d-2722eaaa0d51 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:28.722257545 +0000 UTC m=+36.823070239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/97b3faa7-090c-4ad2-975d-2722eaaa0d51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bwssq" (UID: "97b3faa7-090c-4ad2-975d-2722eaaa0d51") : secret "insights-runtime-extractor-tls" not found Sep 16 13:31:28.222387 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:28.222304 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97b3faa7-090c-4ad2-975d-2722eaaa0d51-kube-rbac-proxy-cm podName:97b3faa7-090c-4ad2-975d-2722eaaa0d51 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:28.722287145 +0000 UTC m=+36.823099858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/97b3faa7-090c-4ad2-975d-2722eaaa0d51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bwssq" (UID: "97b3faa7-090c-4ad2-975d-2722eaaa0d51") : configmap "kube-rbac-proxy" not found Sep 16 13:31:28.222387 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.222351 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/97b3faa7-090c-4ad2-975d-2722eaaa0d51-data-volume\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:28.222580 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.222412 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97b3faa7-090c-4ad2-975d-2722eaaa0d51-crio-socket\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:28.234951 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.234923 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9l99\" (UniqueName: \"kubernetes.io/projected/97b3faa7-090c-4ad2-975d-2722eaaa0d51-kube-api-access-w9l99\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:28.260144 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.260109 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:28.323436 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.323397 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:28.408280 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.408204 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:28.418660 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.418624 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-rr4nd"] Sep 16 13:31:28.437894 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.437862 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rr4nd" Sep 16 13:31:28.441627 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.441578 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-k5z8d\"" Sep 16 13:31:28.442010 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.441840 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 16 13:31:28.442720 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.442697 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-rr4nd"] Sep 16 13:31:28.442843 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.442744 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 16 13:31:28.511294 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.511182 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:28.532172 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.532138 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdh8v\" (UniqueName: \"kubernetes.io/projected/8e942247-6cb7-4e44-82a6-bc7b9c2cf325-kube-api-access-hdh8v\") pod \"migrator-8b4d697c5-rr4nd\" (UID: \"8e942247-6cb7-4e44-82a6-bc7b9c2cf325\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rr4nd" Sep 16 13:31:28.611645 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.611589 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:28.632907 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.632872 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdh8v\" (UniqueName: \"kubernetes.io/projected/8e942247-6cb7-4e44-82a6-bc7b9c2cf325-kube-api-access-hdh8v\") pod \"migrator-8b4d697c5-rr4nd\" (UID: \"8e942247-6cb7-4e44-82a6-bc7b9c2cf325\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rr4nd" Sep 16 13:31:28.642234 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.642205 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdh8v\" (UniqueName: \"kubernetes.io/projected/8e942247-6cb7-4e44-82a6-bc7b9c2cf325-kube-api-access-hdh8v\") pod \"migrator-8b4d697c5-rr4nd\" (UID: \"8e942247-6cb7-4e44-82a6-bc7b9c2cf325\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rr4nd" Sep 16 13:31:28.715134 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.715105 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:28.733476 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.733438 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97b3faa7-090c-4ad2-975d-2722eaaa0d51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:28.733686 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.733512 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97b3faa7-090c-4ad2-975d-2722eaaa0d51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:28.733686 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:28.733619 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 13:31:28.733686 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:28.733687 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97b3faa7-090c-4ad2-975d-2722eaaa0d51-kube-rbac-proxy-cm podName:97b3faa7-090c-4ad2-975d-2722eaaa0d51 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:29.73366859 +0000 UTC m=+37.834481304 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/97b3faa7-090c-4ad2-975d-2722eaaa0d51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bwssq" (UID: "97b3faa7-090c-4ad2-975d-2722eaaa0d51") : configmap "kube-rbac-proxy" not found Sep 16 13:31:28.734042 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:28.734025 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:31:28.734101 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:28.734070 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97b3faa7-090c-4ad2-975d-2722eaaa0d51-insights-runtime-extractor-tls podName:97b3faa7-090c-4ad2-975d-2722eaaa0d51 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:29.734059196 +0000 UTC m=+37.834871901 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/97b3faa7-090c-4ad2-975d-2722eaaa0d51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bwssq" (UID: "97b3faa7-090c-4ad2-975d-2722eaaa0d51") : secret "insights-runtime-extractor-tls" not found Sep 16 13:31:28.748979 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.748950 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rr4nd" Sep 16 13:31:28.966869 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:28.966819 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:29.066159 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:29.066075 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:29.742736 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:29.742699 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97b3faa7-090c-4ad2-975d-2722eaaa0d51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:29.742926 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:29.742759 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97b3faa7-090c-4ad2-975d-2722eaaa0d51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:29.742926 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:29.742854 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 13:31:29.742926 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:29.742896 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:31:29.742926 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:29.742923 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97b3faa7-090c-4ad2-975d-2722eaaa0d51-kube-rbac-proxy-cm podName:97b3faa7-090c-4ad2-975d-2722eaaa0d51 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:31.742905024 +0000 UTC m=+39.843717733 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/97b3faa7-090c-4ad2-975d-2722eaaa0d51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bwssq" (UID: "97b3faa7-090c-4ad2-975d-2722eaaa0d51") : configmap "kube-rbac-proxy" not found Sep 16 13:31:29.743092 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:29.742963 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97b3faa7-090c-4ad2-975d-2722eaaa0d51-insights-runtime-extractor-tls podName:97b3faa7-090c-4ad2-975d-2722eaaa0d51 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:31.742944576 +0000 UTC m=+39.843757284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/97b3faa7-090c-4ad2-975d-2722eaaa0d51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bwssq" (UID: "97b3faa7-090c-4ad2-975d-2722eaaa0d51") : secret "insights-runtime-extractor-tls" not found Sep 16 13:31:29.871166 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:29.871139 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mf77j_ab189943-4b62-4e77-87bd-75934dcb19cb/dns-node-resolver/0.log" Sep 16 13:31:30.045670 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:30.045551 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb307e0c-d166-40bb-ae14-2315eb27a09c-metrics-tls\") pod \"dns-default-jb79h\" (UID: \"bb307e0c-d166-40bb-ae14-2315eb27a09c\") " pod="openshift-dns/dns-default-jb79h" Sep 16 13:31:30.045820 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:30.045733 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:31:30.045820 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:30.045815 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb307e0c-d166-40bb-ae14-2315eb27a09c-metrics-tls podName:bb307e0c-d166-40bb-ae14-2315eb27a09c nodeName:}" failed. No retries permitted until 2026-09-16 13:31:34.045792506 +0000 UTC m=+42.146605203 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb307e0c-d166-40bb-ae14-2315eb27a09c-metrics-tls") pod "dns-default-jb79h" (UID: "bb307e0c-d166-40bb-ae14-2315eb27a09c") : secret "dns-default-metrics-tls" not found Sep 16 13:31:30.140891 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:30.140848 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:30.589942 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:30.589900 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:30.650002 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:30.649949 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:31.073351 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:31.073124 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mslhp_3b1c10dc-eefa-4040-91e6-c52ce36a7ac5/node-ca/0.log" Sep 16 13:31:31.132508 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:31.132478 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-rr4nd"] Sep 16 13:31:31.166709 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:31:31.166677 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e942247_6cb7_4e44_82a6_bc7b9c2cf325.slice/crio-1ca0203366260705bf4d00481bc4fa33441b3d444b229b71e2420893ec3a596c WatchSource:0}: Error finding container 1ca0203366260705bf4d00481bc4fa33441b3d444b229b71e2420893ec3a596c: Status 404 returned error can't find the container with id 1ca0203366260705bf4d00481bc4fa33441b3d444b229b71e2420893ec3a596c Sep 16 13:31:31.585707 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:31.585671 2580 generic.go:356] "Generic (PLEG): container finished" podID="70b7167e-0042-4b50-905c-2cb444c08bc3" containerID="57bdaa6cc36d66a911877826835ba3d49a753e8a54efbe0585cab6c166207769" exitCode=0 Sep 16 13:31:31.585871 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:31.585727 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wml9m" event={"ID":"70b7167e-0042-4b50-905c-2cb444c08bc3","Type":"ContainerDied","Data":"57bdaa6cc36d66a911877826835ba3d49a753e8a54efbe0585cab6c166207769"} Sep 16 13:31:31.587293 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:31.587267 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fr876" event={"ID":"b1b0a620-4bae-4969-916f-3c5019de5654","Type":"ContainerStarted","Data":"7e6fb53522a8d4ca90f6b2cb177701f45bee0680066dae870047a777dcfb818f"} Sep 16 13:31:31.588376 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:31.588354 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rr4nd" event={"ID":"8e942247-6cb7-4e44-82a6-bc7b9c2cf325","Type":"ContainerStarted","Data":"1ca0203366260705bf4d00481bc4fa33441b3d444b229b71e2420893ec3a596c"} Sep 16 13:31:31.638279 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:31.638034 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fr876" podStartSLOduration=1.736134383 podStartE2EDuration="5.638010124s" totalImagesPullingTime="3.901875741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:31.637215752 +0000 UTC m=+39.738028467" watchObservedRunningTime="2026-09-16 13:31:31.638010124 +0000 UTC m=+39.738822844" Sep 16 13:31:31.686829 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:31.686798 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:31.759217 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:31.759193 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97b3faa7-090c-4ad2-975d-2722eaaa0d51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:31.759321 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:31.759237 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97b3faa7-090c-4ad2-975d-2722eaaa0d51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:31.759359 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:31.759345 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:31:31.759426 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:31.759414 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97b3faa7-090c-4ad2-975d-2722eaaa0d51-insights-runtime-extractor-tls podName:97b3faa7-090c-4ad2-975d-2722eaaa0d51 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:35.759396289 +0000 UTC m=+43.860208982 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/97b3faa7-090c-4ad2-975d-2722eaaa0d51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bwssq" (UID: "97b3faa7-090c-4ad2-975d-2722eaaa0d51") : secret "insights-runtime-extractor-tls" not found Sep 16 13:31:31.759748 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:31.759732 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97b3faa7-090c-4ad2-975d-2722eaaa0d51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:32.229864 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:32.229833 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:32.318450 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:32.318413 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:32.595713 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:32.595636 2580 generic.go:356] "Generic (PLEG): container finished" podID="70b7167e-0042-4b50-905c-2cb444c08bc3" containerID="03e626e4c6b8a658d40a27e3bfd8a0a2927f5dbd0672da63f2c791a062c590a7" exitCode=0 Sep 16 13:31:32.595850 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:32.595711 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wml9m" event={"ID":"70b7167e-0042-4b50-905c-2cb444c08bc3","Type":"ContainerDied","Data":"03e626e4c6b8a658d40a27e3bfd8a0a2927f5dbd0672da63f2c791a062c590a7"} Sep 16 13:31:33.285419 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:33.285388 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:33.386094 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:33.386070 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:33.601434 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:33.601399 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wml9m" event={"ID":"70b7167e-0042-4b50-905c-2cb444c08bc3","Type":"ContainerStarted","Data":"474955a72ffabe399f8fc90a9c8b83384ace84fdbf3dded8f7e60431f75a1bcf"} Sep 16 13:31:33.603193 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:33.603161 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rr4nd" event={"ID":"8e942247-6cb7-4e44-82a6-bc7b9c2cf325","Type":"ContainerStarted","Data":"d99ff5383baf92de9d0a9f78ae14d3e9ff3f5bcda51c96bbdf5e203ba6808c0e"} Sep 16 13:31:33.603301 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:33.603218 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rr4nd" event={"ID":"8e942247-6cb7-4e44-82a6-bc7b9c2cf325","Type":"ContainerStarted","Data":"355e2c98857eaf6f841712e81b42b00fe85a7bccb96b6bbdf12cd055c38557ef"} Sep 16 13:31:33.640091 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:33.640042 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wml9m" podStartSLOduration=8.729643433 podStartE2EDuration="41.640026693s" totalImagesPullingTime="32.91038326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:30:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:33.63820342 +0000 UTC m=+41.739016135" watchObservedRunningTime="2026-09-16 13:31:33.640026693 +0000 UTC m=+41.740839408" Sep 16 13:31:33.656809 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:33.656757 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rr4nd" podStartSLOduration=3.636189115 podStartE2EDuration="5.65674675s" totalImagesPullingTime="2.020557635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:33.656108792 +0000 UTC m=+41.756921517" watchObservedRunningTime="2026-09-16 13:31:33.65674675 +0000 UTC m=+41.757559502" Sep 16 13:31:34.077355 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:34.077312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb307e0c-d166-40bb-ae14-2315eb27a09c-metrics-tls\") pod \"dns-default-jb79h\" (UID: \"bb307e0c-d166-40bb-ae14-2315eb27a09c\") " pod="openshift-dns/dns-default-jb79h" Sep 16 13:31:34.077510 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:34.077428 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:31:34.077510 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:34.077481 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb307e0c-d166-40bb-ae14-2315eb27a09c-metrics-tls podName:bb307e0c-d166-40bb-ae14-2315eb27a09c nodeName:}" failed. No retries permitted until 2026-09-16 13:31:42.077468764 +0000 UTC m=+50.178281457 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb307e0c-d166-40bb-ae14-2315eb27a09c-metrics-tls") pod "dns-default-jb79h" (UID: "bb307e0c-d166-40bb-ae14-2315eb27a09c") : secret "dns-default-metrics-tls" not found Sep 16 13:31:35.370409 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:35.370370 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:35.788825 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:35.788789 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97b3faa7-090c-4ad2-975d-2722eaaa0d51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:35.788993 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:35.788944 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:31:35.789032 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:35.789006 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97b3faa7-090c-4ad2-975d-2722eaaa0d51-insights-runtime-extractor-tls podName:97b3faa7-090c-4ad2-975d-2722eaaa0d51 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:43.788988853 +0000 UTC m=+51.889801571 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/97b3faa7-090c-4ad2-975d-2722eaaa0d51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bwssq" (UID: "97b3faa7-090c-4ad2-975d-2722eaaa0d51") : secret "insights-runtime-extractor-tls" not found Sep 16 13:31:35.873598 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:35.873556 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:35.976544 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:35.976499 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:36.078282 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:36.078205 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:36.264621 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:36.264565 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:36.357401 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:36.357315 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:36.491621 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:36.491577 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:37.098981 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:37.098932 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:37.685864 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:37.685829 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:37.761233 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:37.761200 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:38.294282 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:38.294254 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:38.495434 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:38.495402 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:38.890020 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:38.889920 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:39.093395 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:39.093344 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:39.499371 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:39.499339 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:39.575736 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:39.575707 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:39.808046 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:39.807969 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:40.296851 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:40.296820 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:40.712436 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:40.712350 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:40.889793 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:40.889758 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:41.107980 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:41.107945 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:41.498868 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:41.498841 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:42.131801 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:42.131762 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb307e0c-d166-40bb-ae14-2315eb27a09c-metrics-tls\") pod \"dns-default-jb79h\" (UID: \"bb307e0c-d166-40bb-ae14-2315eb27a09c\") " pod="openshift-dns/dns-default-jb79h" Sep 16 13:31:42.135098 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:42.135078 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb307e0c-d166-40bb-ae14-2315eb27a09c-metrics-tls\") pod \"dns-default-jb79h\" (UID: \"bb307e0c-d166-40bb-ae14-2315eb27a09c\") " pod="openshift-dns/dns-default-jb79h" Sep 16 13:31:42.216742 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:42.216694 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jb79h" Sep 16 13:31:42.490159 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:42.490127 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jb79h"] Sep 16 13:31:42.493696 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:31:42.493667 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb307e0c_d166_40bb_ae14_2315eb27a09c.slice/crio-dd361e589900d93f75f7d64ac8d06076f6cad334411a1f6ec631a6bd5b7ba966 WatchSource:0}: Error finding container dd361e589900d93f75f7d64ac8d06076f6cad334411a1f6ec631a6bd5b7ba966: Status 404 returned error can't find the container with id dd361e589900d93f75f7d64ac8d06076f6cad334411a1f6ec631a6bd5b7ba966 Sep 16 13:31:42.600057 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:42.600020 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:42.626166 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:42.625945 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jb79h" event={"ID":"bb307e0c-d166-40bb-ae14-2315eb27a09c","Type":"ContainerStarted","Data":"dd361e589900d93f75f7d64ac8d06076f6cad334411a1f6ec631a6bd5b7ba966"} Sep 16 13:31:42.705290 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:42.705255 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:42.896469 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:42.896372 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:43.087780 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:43.087728 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:43.489446 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:43.489402 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:43.844206 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:43.844095 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97b3faa7-090c-4ad2-975d-2722eaaa0d51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:43.846919 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:43.846888 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97b3faa7-090c-4ad2-975d-2722eaaa0d51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bwssq\" (UID: \"97b3faa7-090c-4ad2-975d-2722eaaa0d51\") " pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:43.925133 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:43.925091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bwssq" Sep 16 13:31:44.199461 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:44.199377 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bwssq"] Sep 16 13:31:44.202933 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:31:44.202905 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b3faa7_090c_4ad2_975d_2722eaaa0d51.slice/crio-ce17a84d728d1385723ca6b22af18cccb5936f26a77d959c0d27777be66a253c WatchSource:0}: Error finding container ce17a84d728d1385723ca6b22af18cccb5936f26a77d959c0d27777be66a253c: Status 404 returned error can't find the container with id ce17a84d728d1385723ca6b22af18cccb5936f26a77d959c0d27777be66a253c Sep 16 13:31:44.570352 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:44.570314 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:44.631663 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:44.631631 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:44.632401 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:44.632371 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bwssq" event={"ID":"97b3faa7-090c-4ad2-975d-2722eaaa0d51","Type":"ContainerStarted","Data":"ac9edcb6ff4d4e6e00c4bede661f052ce9ab912a7f251233bd0cb72a10620a0f"} Sep 16 13:31:44.632513 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:44.632440 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bwssq" event={"ID":"97b3faa7-090c-4ad2-975d-2722eaaa0d51","Type":"ContainerStarted","Data":"ce17a84d728d1385723ca6b22af18cccb5936f26a77d959c0d27777be66a253c"} Sep 16 13:31:44.934200 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:44.934170 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:45.107584 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:45.107497 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:45.490091 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:45.490048 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:45.637998 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:45.637962 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jb79h" event={"ID":"bb307e0c-d166-40bb-ae14-2315eb27a09c","Type":"ContainerStarted","Data":"feaaf93158315d41dcdddaf2b18fc10bfb0cfb1bead27c830112101d17aa6ac6"} Sep 16 13:31:45.638169 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:45.638021 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jb79h" event={"ID":"bb307e0c-d166-40bb-ae14-2315eb27a09c","Type":"ContainerStarted","Data":"39b357d2e1984a3659484c564e0db17abb61f6b08746676e664b9d9c791fcd1f"} Sep 16 13:31:45.638169 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:45.638080 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jb79h" Sep 16 13:31:45.657891 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:45.657843 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jb79h" podStartSLOduration=17.334768911 podStartE2EDuration="19.657827473s" totalImagesPullingTime="2.323058562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:45.657149073 +0000 UTC m=+53.757961787" watchObservedRunningTime="2026-09-16 13:31:45.657827473 +0000 UTC m=+53.758640187" Sep 16 13:31:46.086828 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:46.086797 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:46.287982 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:46.287944 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:46.550242 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:46.550162 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:46.645509 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:46.645461 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:46.647576 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:46.647536 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bwssq" event={"ID":"97b3faa7-090c-4ad2-975d-2722eaaa0d51","Type":"ContainerStarted","Data":"c32cd058ae557dc990709fa505bf2e17e6c2d127c42c6218be0c92cf83e9d151"} Sep 16 13:31:46.791926 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:46.791893 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:46.981172 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:46.981136 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:47.186140 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:47.186106 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:47.274164 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:47.274081 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:47.579660 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:47.579576 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:47.652455 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:47.652416 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bwssq" event={"ID":"97b3faa7-090c-4ad2-975d-2722eaaa0d51","Type":"ContainerStarted","Data":"65516a047ba938863ecc3ec1b1092dc4449480360f6a81926f547fe81483063e"} Sep 16 13:31:47.684314 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:47.684260 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bwssq" podStartSLOduration=17.932823844 podStartE2EDuration="20.684243391s" totalImagesPullingTime="2.751419547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:31:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:47.683725192 +0000 UTC m=+55.784537910" watchObservedRunningTime="2026-09-16 13:31:47.684243391 +0000 UTC m=+55.785056105" Sep 16 13:31:47.789171 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:47.789131 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:48.186616 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:48.186584 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:48.288009 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:48.287976 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:48.388438 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:48.388409 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:48.597974 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:48.597944 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:48.698481 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:48.698451 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:48.765554 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:48.765524 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:53.205256 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:53.205225 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:53.306244 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:53.306209 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:53.570863 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:53.570838 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4l8qb" Sep 16 13:31:54.166048 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.166015 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc"] Sep 16 13:31:54.168897 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.168875 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" Sep 16 13:31:54.172224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.172032 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 16 13:31:54.172224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.172130 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 16 13:31:54.172224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.172152 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 16 13:31:54.173031 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.173009 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 16 13:31:54.173169 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.173035 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 16 13:31:54.173169 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.173099 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 16 13:31:54.173468 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.173449 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 16 13:31:54.193591 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.193565 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc"] Sep 16 13:31:54.316100 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.316065 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/96493325-bc47-479d-9fee-6a19b7e423e9-hub\") pod \"cluster-proxy-proxy-agent-8696b668d9-nk8cc\" (UID: \"96493325-bc47-479d-9fee-6a19b7e423e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" Sep 16 13:31:54.316100 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.316106 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/96493325-bc47-479d-9fee-6a19b7e423e9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8696b668d9-nk8cc\" (UID: \"96493325-bc47-479d-9fee-6a19b7e423e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" Sep 16 13:31:54.316524 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.316148 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/96493325-bc47-479d-9fee-6a19b7e423e9-ca\") pod \"cluster-proxy-proxy-agent-8696b668d9-nk8cc\" (UID: \"96493325-bc47-479d-9fee-6a19b7e423e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" Sep 16 13:31:54.316524 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.316191 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvkwh\" (UniqueName: \"kubernetes.io/projected/96493325-bc47-479d-9fee-6a19b7e423e9-kube-api-access-qvkwh\") pod \"cluster-proxy-proxy-agent-8696b668d9-nk8cc\" (UID: \"96493325-bc47-479d-9fee-6a19b7e423e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" Sep 16 13:31:54.316524 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.316268 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/96493325-bc47-479d-9fee-6a19b7e423e9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8696b668d9-nk8cc\" (UID: \"96493325-bc47-479d-9fee-6a19b7e423e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" Sep 16 13:31:54.316524 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.316294 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/96493325-bc47-479d-9fee-6a19b7e423e9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8696b668d9-nk8cc\" (UID: \"96493325-bc47-479d-9fee-6a19b7e423e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" Sep 16 13:31:54.326396 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.326369 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b544f48c8-lmbfw"] Sep 16 13:31:54.328131 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.328116 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.332277 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.332254 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 16 13:31:54.332393 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.332379 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 16 13:31:54.332620 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.332593 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 16 13:31:54.335711 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.335694 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s4fpm\"" Sep 16 13:31:54.353426 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.353403 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 16 13:31:54.362473 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.362452 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b544f48c8-lmbfw"] Sep 16 13:31:54.369401 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.369381 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:54.411475 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.411442 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b544f48c8-lmbfw"] Sep 16 13:31:54.411680 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:31:54.411658 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-9h8p8 registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" podUID="00ba1faa-2848-4f65-b12b-5d6dbae53092" Sep 16 13:31:54.416762 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.416716 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00ba1faa-2848-4f65-b12b-5d6dbae53092-registry-certificates\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.416762 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.416752 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/96493325-bc47-479d-9fee-6a19b7e423e9-hub\") pod \"cluster-proxy-proxy-agent-8696b668d9-nk8cc\" (UID: \"96493325-bc47-479d-9fee-6a19b7e423e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" Sep 16 13:31:54.416898 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.416776 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/96493325-bc47-479d-9fee-6a19b7e423e9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8696b668d9-nk8cc\" (UID: \"96493325-bc47-479d-9fee-6a19b7e423e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" Sep 16 13:31:54.416898 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.416804 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/96493325-bc47-479d-9fee-6a19b7e423e9-ca\") pod \"cluster-proxy-proxy-agent-8696b668d9-nk8cc\" (UID: \"96493325-bc47-479d-9fee-6a19b7e423e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" Sep 16 13:31:54.416898 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.416844 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00ba1faa-2848-4f65-b12b-5d6dbae53092-image-registry-private-configuration\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.416898 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.416887 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00ba1faa-2848-4f65-b12b-5d6dbae53092-ca-trust-extracted\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.417073 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.416914 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00ba1faa-2848-4f65-b12b-5d6dbae53092-trusted-ca\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.417073 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.416965 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00ba1faa-2848-4f65-b12b-5d6dbae53092-bound-sa-token\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.417073 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.417043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvkwh\" (UniqueName: \"kubernetes.io/projected/96493325-bc47-479d-9fee-6a19b7e423e9-kube-api-access-qvkwh\") pod \"cluster-proxy-proxy-agent-8696b668d9-nk8cc\" (UID: \"96493325-bc47-479d-9fee-6a19b7e423e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" Sep 16 13:31:54.417219 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.417096 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00ba1faa-2848-4f65-b12b-5d6dbae53092-registry-tls\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.417219 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.417135 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/96493325-bc47-479d-9fee-6a19b7e423e9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8696b668d9-nk8cc\" (UID: \"96493325-bc47-479d-9fee-6a19b7e423e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" Sep 16 13:31:54.417319 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.417190 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/96493325-bc47-479d-9fee-6a19b7e423e9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8696b668d9-nk8cc\" (UID: \"96493325-bc47-479d-9fee-6a19b7e423e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" Sep 16 13:31:54.417319 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.417250 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00ba1faa-2848-4f65-b12b-5d6dbae53092-installation-pull-secrets\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.417422 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.417348 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h8p8\" (UniqueName: \"kubernetes.io/projected/00ba1faa-2848-4f65-b12b-5d6dbae53092-kube-api-access-9h8p8\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.417894 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.417851 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/96493325-bc47-479d-9fee-6a19b7e423e9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8696b668d9-nk8cc\" (UID: \"96493325-bc47-479d-9fee-6a19b7e423e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" Sep 16 13:31:54.419554 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.419528 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/96493325-bc47-479d-9fee-6a19b7e423e9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8696b668d9-nk8cc\" (UID: \"96493325-bc47-479d-9fee-6a19b7e423e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" Sep 16 13:31:54.419693 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.419565 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/96493325-bc47-479d-9fee-6a19b7e423e9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8696b668d9-nk8cc\" (UID: \"96493325-bc47-479d-9fee-6a19b7e423e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" Sep 16 13:31:54.419776 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.419720 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/96493325-bc47-479d-9fee-6a19b7e423e9-ca\") pod \"cluster-proxy-proxy-agent-8696b668d9-nk8cc\" (UID: \"96493325-bc47-479d-9fee-6a19b7e423e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" Sep 16 13:31:54.419930 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.419910 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/96493325-bc47-479d-9fee-6a19b7e423e9-hub\") pod \"cluster-proxy-proxy-agent-8696b668d9-nk8cc\" (UID: \"96493325-bc47-479d-9fee-6a19b7e423e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" Sep 16 13:31:54.434485 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.434462 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvkwh\" (UniqueName: \"kubernetes.io/projected/96493325-bc47-479d-9fee-6a19b7e423e9-kube-api-access-qvkwh\") pod \"cluster-proxy-proxy-agent-8696b668d9-nk8cc\" (UID: \"96493325-bc47-479d-9fee-6a19b7e423e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" Sep 16 13:31:54.488023 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.487987 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" Sep 16 13:31:54.519938 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.517789 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00ba1faa-2848-4f65-b12b-5d6dbae53092-image-registry-private-configuration\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.519938 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.517837 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00ba1faa-2848-4f65-b12b-5d6dbae53092-ca-trust-extracted\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.519938 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.517863 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00ba1faa-2848-4f65-b12b-5d6dbae53092-trusted-ca\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.519938 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.517888 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00ba1faa-2848-4f65-b12b-5d6dbae53092-bound-sa-token\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.519938 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.517938 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00ba1faa-2848-4f65-b12b-5d6dbae53092-registry-tls\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.519938 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.517976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00ba1faa-2848-4f65-b12b-5d6dbae53092-installation-pull-secrets\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.519938 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.518013 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h8p8\" (UniqueName: \"kubernetes.io/projected/00ba1faa-2848-4f65-b12b-5d6dbae53092-kube-api-access-9h8p8\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.519938 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.518040 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00ba1faa-2848-4f65-b12b-5d6dbae53092-registry-certificates\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.519938 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.518936 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00ba1faa-2848-4f65-b12b-5d6dbae53092-registry-certificates\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.519938 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.519874 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00ba1faa-2848-4f65-b12b-5d6dbae53092-ca-trust-extracted\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.521088 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.520826 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00ba1faa-2848-4f65-b12b-5d6dbae53092-trusted-ca\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.522252 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.522205 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00ba1faa-2848-4f65-b12b-5d6dbae53092-image-registry-private-configuration\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.523015 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.522995 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00ba1faa-2848-4f65-b12b-5d6dbae53092-registry-tls\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.524787 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.524752 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00ba1faa-2848-4f65-b12b-5d6dbae53092-installation-pull-secrets\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.530417 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.530390 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00ba1faa-2848-4f65-b12b-5d6dbae53092-bound-sa-token\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.531689 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.531631 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h8p8\" (UniqueName: \"kubernetes.io/projected/00ba1faa-2848-4f65-b12b-5d6dbae53092-kube-api-access-9h8p8\") pod \"image-registry-6b544f48c8-lmbfw\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.574183 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.574146 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:54.673917 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.673832 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.674946 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.674916 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:54.680181 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.680161 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:54.769077 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.769046 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc"] Sep 16 13:31:54.772120 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:31:54.772097 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96493325_bc47_479d_9fee_6a19b7e423e9.slice/crio-4cc344bd39c835d42458b7223f4f2ea8499e087fc1681ab500647a1de79606c9 WatchSource:0}: Error finding container 4cc344bd39c835d42458b7223f4f2ea8499e087fc1681ab500647a1de79606c9: Status 404 returned error can't find the container with id 4cc344bd39c835d42458b7223f4f2ea8499e087fc1681ab500647a1de79606c9 Sep 16 13:31:54.819487 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.819459 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00ba1faa-2848-4f65-b12b-5d6dbae53092-registry-certificates\") pod \"00ba1faa-2848-4f65-b12b-5d6dbae53092\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " Sep 16 13:31:54.819692 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.819489 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00ba1faa-2848-4f65-b12b-5d6dbae53092-ca-trust-extracted\") pod \"00ba1faa-2848-4f65-b12b-5d6dbae53092\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " Sep 16 13:31:54.819692 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.819522 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00ba1faa-2848-4f65-b12b-5d6dbae53092-installation-pull-secrets\") pod \"00ba1faa-2848-4f65-b12b-5d6dbae53092\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " Sep 16 13:31:54.819692 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.819544 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00ba1faa-2848-4f65-b12b-5d6dbae53092-bound-sa-token\") pod \"00ba1faa-2848-4f65-b12b-5d6dbae53092\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " Sep 16 13:31:54.819692 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.819559 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h8p8\" (UniqueName: \"kubernetes.io/projected/00ba1faa-2848-4f65-b12b-5d6dbae53092-kube-api-access-9h8p8\") pod \"00ba1faa-2848-4f65-b12b-5d6dbae53092\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " Sep 16 13:31:54.819692 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.819582 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00ba1faa-2848-4f65-b12b-5d6dbae53092-registry-tls\") pod \"00ba1faa-2848-4f65-b12b-5d6dbae53092\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " Sep 16 13:31:54.819692 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.819648 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00ba1faa-2848-4f65-b12b-5d6dbae53092-image-registry-private-configuration\") pod \"00ba1faa-2848-4f65-b12b-5d6dbae53092\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " Sep 16 13:31:54.819692 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.819667 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00ba1faa-2848-4f65-b12b-5d6dbae53092-trusted-ca\") pod \"00ba1faa-2848-4f65-b12b-5d6dbae53092\" (UID: \"00ba1faa-2848-4f65-b12b-5d6dbae53092\") " Sep 16 13:31:54.820033 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.819826 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ba1faa-2848-4f65-b12b-5d6dbae53092-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "00ba1faa-2848-4f65-b12b-5d6dbae53092" (UID: "00ba1faa-2848-4f65-b12b-5d6dbae53092"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:31:54.820033 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.819926 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00ba1faa-2848-4f65-b12b-5d6dbae53092-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "00ba1faa-2848-4f65-b12b-5d6dbae53092" (UID: "00ba1faa-2848-4f65-b12b-5d6dbae53092"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:31:54.820232 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.820206 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00ba1faa-2848-4f65-b12b-5d6dbae53092-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "00ba1faa-2848-4f65-b12b-5d6dbae53092" (UID: "00ba1faa-2848-4f65-b12b-5d6dbae53092"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:31:54.822147 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.822118 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00ba1faa-2848-4f65-b12b-5d6dbae53092-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "00ba1faa-2848-4f65-b12b-5d6dbae53092" (UID: "00ba1faa-2848-4f65-b12b-5d6dbae53092"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:31:54.822454 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.822431 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00ba1faa-2848-4f65-b12b-5d6dbae53092-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "00ba1faa-2848-4f65-b12b-5d6dbae53092" (UID: "00ba1faa-2848-4f65-b12b-5d6dbae53092"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:31:54.822522 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.822496 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ba1faa-2848-4f65-b12b-5d6dbae53092-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "00ba1faa-2848-4f65-b12b-5d6dbae53092" (UID: "00ba1faa-2848-4f65-b12b-5d6dbae53092"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:31:54.822522 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.822497 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ba1faa-2848-4f65-b12b-5d6dbae53092-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "00ba1faa-2848-4f65-b12b-5d6dbae53092" (UID: "00ba1faa-2848-4f65-b12b-5d6dbae53092"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:31:54.822522 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.822510 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ba1faa-2848-4f65-b12b-5d6dbae53092-kube-api-access-9h8p8" (OuterVolumeSpecName: "kube-api-access-9h8p8") pod "00ba1faa-2848-4f65-b12b-5d6dbae53092" (UID: "00ba1faa-2848-4f65-b12b-5d6dbae53092"). InnerVolumeSpecName "kube-api-access-9h8p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:31:54.914737 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.914698 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:54.920927 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.920906 2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00ba1faa-2848-4f65-b12b-5d6dbae53092-registry-certificates\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:31:54.921016 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.920929 2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00ba1faa-2848-4f65-b12b-5d6dbae53092-ca-trust-extracted\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:31:54.921016 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.920945 2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00ba1faa-2848-4f65-b12b-5d6dbae53092-installation-pull-secrets\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:31:54.921016 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.920956 2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00ba1faa-2848-4f65-b12b-5d6dbae53092-bound-sa-token\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:31:54.921016 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.920965 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h8p8\" (UniqueName: \"kubernetes.io/projected/00ba1faa-2848-4f65-b12b-5d6dbae53092-kube-api-access-9h8p8\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:31:54.921016 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.920973 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00ba1faa-2848-4f65-b12b-5d6dbae53092-registry-tls\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:31:54.921016 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.920981 2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00ba1faa-2848-4f65-b12b-5d6dbae53092-image-registry-private-configuration\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:31:54.921016 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.920989 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00ba1faa-2848-4f65-b12b-5d6dbae53092-trusted-ca\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:31:54.984711 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:54.984680 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:55.650172 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:55.650141 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jb79h" Sep 16 13:31:55.677577 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:55.677551 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b544f48c8-lmbfw" Sep 16 13:31:55.677730 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:55.677576 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" event={"ID":"96493325-bc47-479d-9fee-6a19b7e423e9","Type":"ContainerStarted","Data":"4cc344bd39c835d42458b7223f4f2ea8499e087fc1681ab500647a1de79606c9"} Sep 16 13:31:55.717088 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:55.717043 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b544f48c8-lmbfw"] Sep 16 13:31:55.726478 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:55.726442 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6b544f48c8-lmbfw"] Sep 16 13:31:55.928288 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:55.928204 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:56.085592 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:56.085537 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:56.225021 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:56.224986 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:56.345920 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:56.345891 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:56.353446 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:56.353411 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00ba1faa-2848-4f65-b12b-5d6dbae53092" path="/var/lib/kubelet/pods/00ba1faa-2848-4f65-b12b-5d6dbae53092/volumes" Sep 16 13:31:56.958314 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:56.958280 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:57.188661 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:57.188622 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:58.047482 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:58.047443 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs\") pod \"network-metrics-daemon-c4xxk\" (UID: \"cd501395-aaf2-4005-95bd-f202a13d61b5\") " pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:31:58.047921 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:58.047515 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret\") pod \"global-pull-secret-syncer-cmvcd\" (UID: \"5f469d5f-6b1f-4f0e-910d-bff21be9a372\") " pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:31:58.050531 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:58.050503 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 13:31:58.051540 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:58.051521 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 13:31:58.060877 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:58.060851 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd501395-aaf2-4005-95bd-f202a13d61b5-metrics-certs\") pod \"network-metrics-daemon-c4xxk\" (UID: \"cd501395-aaf2-4005-95bd-f202a13d61b5\") " pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:31:58.061002 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:58.060946 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f469d5f-6b1f-4f0e-910d-bff21be9a372-original-pull-secret\") pod \"global-pull-secret-syncer-cmvcd\" (UID: \"5f469d5f-6b1f-4f0e-910d-bff21be9a372\") " pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:31:58.167669 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:58.167628 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmvcd" Sep 16 13:31:58.176771 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:58.176748 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4n8lp\"" Sep 16 13:31:58.183866 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:58.183843 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4xxk" Sep 16 13:31:58.249681 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:58.249645 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjpw\" (UniqueName: \"kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw\") pod \"network-check-target-m6j5f\" (UID: \"b313af30-1af8-4dac-a6f6-fa42450ea3df\") " pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:31:58.253251 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:58.253228 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 13:31:58.263188 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:58.263166 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 13:31:58.273424 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:58.273397 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wjpw\" (UniqueName: \"kubernetes.io/projected/b313af30-1af8-4dac-a6f6-fa42450ea3df-kube-api-access-2wjpw\") pod \"network-check-target-m6j5f\" (UID: \"b313af30-1af8-4dac-a6f6-fa42450ea3df\") " pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:31:58.464209 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:58.464119 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4nxp8\"" Sep 16 13:31:58.471830 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:58.471802 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:31:58.690376 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:58.690347 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" event={"ID":"96493325-bc47-479d-9fee-6a19b7e423e9","Type":"ContainerStarted","Data":"b1008d7d3b3f9725c9c43ec742e0c59cf67c2dd9a3b7a65dbc05d9ce272df3dc"} Sep 16 13:31:58.690496 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:58.690395 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" event={"ID":"96493325-bc47-479d-9fee-6a19b7e423e9","Type":"ContainerStarted","Data":"3f2b2de673094da46cec018b3d5cec479e4a4a7c97122542676082196bd4bf2d"} Sep 16 13:31:58.744843 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:58.744768 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cmvcd"] Sep 16 13:31:58.748041 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:31:58.748011 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f469d5f_6b1f_4f0e_910d_bff21be9a372.slice/crio-210c47ca64915710f1570fda98e444034b2ec96c6a15191fc9617c8c638dcbbc WatchSource:0}: Error finding container 210c47ca64915710f1570fda98e444034b2ec96c6a15191fc9617c8c638dcbbc: Status 404 returned error can't find the container with id 210c47ca64915710f1570fda98e444034b2ec96c6a15191fc9617c8c638dcbbc Sep 16 13:31:58.750042 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:58.750020 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c4xxk"] Sep 16 13:31:58.753701 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:31:58.753678 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd501395_aaf2_4005_95bd_f202a13d61b5.slice/crio-2352481342ad00995b656d8f1b710ba8d723fcd56f8a366470ffad28d0917fc2 WatchSource:0}: Error finding container 2352481342ad00995b656d8f1b710ba8d723fcd56f8a366470ffad28d0917fc2: Status 404 returned error can't find the container with id 2352481342ad00995b656d8f1b710ba8d723fcd56f8a366470ffad28d0917fc2 Sep 16 13:31:58.777206 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:58.777172 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m6j5f"] Sep 16 13:31:58.780145 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:31:58.780122 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb313af30_1af8_4dac_a6f6_fa42450ea3df.slice/crio-cc01e18817d38882534cb76bb5ff6f6c825eb1424e40edf0c53670fb313802db WatchSource:0}: Error finding container cc01e18817d38882534cb76bb5ff6f6c825eb1424e40edf0c53670fb313802db: Status 404 returned error can't find the container with id cc01e18817d38882534cb76bb5ff6f6c825eb1424e40edf0c53670fb313802db Sep 16 13:31:58.900801 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:58.900765 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:59.007191 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:59.007111 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:59.695389 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:59.695350 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c4xxk" event={"ID":"cd501395-aaf2-4005-95bd-f202a13d61b5","Type":"ContainerStarted","Data":"2352481342ad00995b656d8f1b710ba8d723fcd56f8a366470ffad28d0917fc2"} Sep 16 13:31:59.698327 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:59.698292 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" event={"ID":"96493325-bc47-479d-9fee-6a19b7e423e9","Type":"ContainerStarted","Data":"a120f7f3e1d8012588e881db2fb6826c4ffee6b3a025198fc8360f662d2a06a9"} Sep 16 13:31:59.699879 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:59.699848 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m6j5f" event={"ID":"b313af30-1af8-4dac-a6f6-fa42450ea3df","Type":"ContainerStarted","Data":"cc01e18817d38882534cb76bb5ff6f6c825eb1424e40edf0c53670fb313802db"} Sep 16 13:31:59.701376 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:59.701346 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cmvcd" event={"ID":"5f469d5f-6b1f-4f0e-910d-bff21be9a372","Type":"ContainerStarted","Data":"210c47ca64915710f1570fda98e444034b2ec96c6a15191fc9617c8c638dcbbc"} Sep 16 13:31:59.722490 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:31:59.722431 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8696b668d9-nk8cc" podStartSLOduration=1.981054544 podStartE2EDuration="5.722414673s" totalImagesPullingTime="3.741360129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:59.720305768 +0000 UTC m=+67.821118483" watchObservedRunningTime="2026-09-16 13:31:59.722414673 +0000 UTC m=+67.823227387" Sep 16 13:32:00.709802 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:00.709713 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c4xxk" event={"ID":"cd501395-aaf2-4005-95bd-f202a13d61b5","Type":"ContainerStarted","Data":"d357986ea83da2dcf177db00e4f1bfe90ac52eb8bfab35d2a8ccca9b65a36033"} Sep 16 13:32:00.709802 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:00.709782 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c4xxk" event={"ID":"cd501395-aaf2-4005-95bd-f202a13d61b5","Type":"ContainerStarted","Data":"e46971bc0a49be91628529fcc61c6f68d4889532dbbfcd1a9f8e36a2a16f7798"} Sep 16 13:32:00.732097 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:00.732035 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-c4xxk" podStartSLOduration=67.349969638 podStartE2EDuration="1m8.732015582s" totalImagesPullingTime="1.382045944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:32:00.731437847 +0000 UTC m=+68.832250562" watchObservedRunningTime="2026-09-16 13:32:00.732015582 +0000 UTC m=+68.832828300" Sep 16 13:32:00.844490 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:00.844456 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:00.957504 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:00.957456 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:03.099009 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:03.098976 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:03.200092 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:03.200054 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:03.333089 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:03.333052 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:03.404335 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:03.404226 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:03.608097 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:03.608063 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:03.709620 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:03.709529 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:04.596760 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:04.596722 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:04.674214 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:04.674183 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:04.728523 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:04.728490 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m6j5f" event={"ID":"b313af30-1af8-4dac-a6f6-fa42450ea3df","Type":"ContainerStarted","Data":"c06ecab7f3c3d91bcceef501190084108ab482e8bf3335dce67c905cd848a88b"} Sep 16 13:32:04.728711 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:04.728584 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:32:04.729915 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:04.729892 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cmvcd" event={"ID":"5f469d5f-6b1f-4f0e-910d-bff21be9a372","Type":"ContainerStarted","Data":"5ecf33a5326ef809645392e5104b9ec983cf6755d5e1b2859d6b2accf86a9fc6"} Sep 16 13:32:04.747064 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:04.747020 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m6j5f" podStartSLOduration=67.453530013 podStartE2EDuration="1m12.747005981s" totalImagesPullingTime="5.293475968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:32:04.746264807 +0000 UTC m=+72.847077525" watchObservedRunningTime="2026-09-16 13:32:04.747005981 +0000 UTC m=+72.847818696" Sep 16 13:32:04.762528 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:04.762484 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cmvcd" podStartSLOduration=67.435595425 podStartE2EDuration="1m12.762469223s" totalImagesPullingTime="5.326873798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:32:04.761821693 +0000 UTC m=+72.862634408" watchObservedRunningTime="2026-09-16 13:32:04.762469223 +0000 UTC m=+72.863281939" Sep 16 13:32:06.610434 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:06.610405 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:06.710914 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:06.710880 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:08.015594 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:08.015559 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:08.153986 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:08.153956 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:08.416082 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:08.416002 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:08.517085 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:08.517063 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:09.816859 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:09.816827 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:09.919222 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:09.919181 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:12.181902 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:12.181868 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:12.246050 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:12.246017 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:12.381760 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:12.381715 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:12.480443 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:12.480412 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:12.793221 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:12.793125 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:12.867988 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:12.867945 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:13.395743 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:13.395537 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:14.513245 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.513213 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:14.715185 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.715154 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:14.809044 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.808966 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:14.812028 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.812004 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb"] Sep 16 13:32:14.814422 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.814407 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" Sep 16 13:32:14.816963 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.816936 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 13:32:14.816963 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.816950 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 13:32:14.816963 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.816961 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 16 13:32:14.818097 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.818077 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 13:32:14.818224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.818103 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 16 13:32:14.818224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.818204 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 16 13:32:14.818224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.818217 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-zgwtk\"" Sep 16 13:32:14.833558 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.833523 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb"] Sep 16 13:32:14.853342 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.853315 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rwsmj"] Sep 16 13:32:14.855402 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.855380 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:14.858528 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.858508 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 13:32:14.858796 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.858778 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 13:32:14.858796 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.858790 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 13:32:14.859968 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.859953 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-txljv\"" Sep 16 13:32:14.863846 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.863825 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/557e9f3c-9f2c-45db-8a69-08a49a6465d2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-q7vzb\" (UID: \"557e9f3c-9f2c-45db-8a69-08a49a6465d2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" Sep 16 13:32:14.863932 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.863859 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/557e9f3c-9f2c-45db-8a69-08a49a6465d2-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-q7vzb\" (UID: \"557e9f3c-9f2c-45db-8a69-08a49a6465d2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" Sep 16 13:32:14.863932 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.863879 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/557e9f3c-9f2c-45db-8a69-08a49a6465d2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-q7vzb\" (UID: \"557e9f3c-9f2c-45db-8a69-08a49a6465d2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" Sep 16 13:32:14.863932 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.863894 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6jht\" (UniqueName: \"kubernetes.io/projected/557e9f3c-9f2c-45db-8a69-08a49a6465d2-kube-api-access-h6jht\") pod \"kube-state-metrics-59fbdd77d-q7vzb\" (UID: \"557e9f3c-9f2c-45db-8a69-08a49a6465d2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" Sep 16 13:32:14.863932 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.863913 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/557e9f3c-9f2c-45db-8a69-08a49a6465d2-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-q7vzb\" (UID: \"557e9f3c-9f2c-45db-8a69-08a49a6465d2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" Sep 16 13:32:14.864105 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.863996 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/557e9f3c-9f2c-45db-8a69-08a49a6465d2-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-q7vzb\" (UID: \"557e9f3c-9f2c-45db-8a69-08a49a6465d2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" Sep 16 13:32:14.965012 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.964970 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/839cb0c4-5760-48c2-87ef-015f82193daf-node-exporter-tls\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:14.965200 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.965023 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6jht\" (UniqueName: \"kubernetes.io/projected/557e9f3c-9f2c-45db-8a69-08a49a6465d2-kube-api-access-h6jht\") pod \"kube-state-metrics-59fbdd77d-q7vzb\" (UID: \"557e9f3c-9f2c-45db-8a69-08a49a6465d2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" Sep 16 13:32:14.965200 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.965047 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/839cb0c4-5760-48c2-87ef-015f82193daf-metrics-client-ca\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:14.965200 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.965094 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/557e9f3c-9f2c-45db-8a69-08a49a6465d2-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-q7vzb\" (UID: \"557e9f3c-9f2c-45db-8a69-08a49a6465d2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" Sep 16 13:32:14.965200 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.965130 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/557e9f3c-9f2c-45db-8a69-08a49a6465d2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-q7vzb\" (UID: \"557e9f3c-9f2c-45db-8a69-08a49a6465d2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" Sep 16 13:32:14.965200 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.965153 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/839cb0c4-5760-48c2-87ef-015f82193daf-node-exporter-accelerators-collector-config\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:14.965200 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.965171 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/557e9f3c-9f2c-45db-8a69-08a49a6465d2-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-q7vzb\" (UID: \"557e9f3c-9f2c-45db-8a69-08a49a6465d2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" Sep 16 13:32:14.965200 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.965190 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/839cb0c4-5760-48c2-87ef-015f82193daf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:14.965534 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.965217 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/839cb0c4-5760-48c2-87ef-015f82193daf-sys\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:14.965534 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.965402 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/557e9f3c-9f2c-45db-8a69-08a49a6465d2-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-q7vzb\" (UID: \"557e9f3c-9f2c-45db-8a69-08a49a6465d2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" Sep 16 13:32:14.965534 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.965455 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/839cb0c4-5760-48c2-87ef-015f82193daf-node-exporter-wtmp\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:14.965534 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.965494 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfgw9\" (UniqueName: \"kubernetes.io/projected/839cb0c4-5760-48c2-87ef-015f82193daf-kube-api-access-wfgw9\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:14.965534 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.965494 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/557e9f3c-9f2c-45db-8a69-08a49a6465d2-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-q7vzb\" (UID: \"557e9f3c-9f2c-45db-8a69-08a49a6465d2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" Sep 16 13:32:14.965534 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:32:14.965530 2580 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 16 13:32:14.965846 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.965535 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/557e9f3c-9f2c-45db-8a69-08a49a6465d2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-q7vzb\" (UID: \"557e9f3c-9f2c-45db-8a69-08a49a6465d2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" Sep 16 13:32:14.965846 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.965565 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/839cb0c4-5760-48c2-87ef-015f82193daf-root\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:14.965846 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:32:14.965632 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/557e9f3c-9f2c-45db-8a69-08a49a6465d2-kube-state-metrics-tls podName:557e9f3c-9f2c-45db-8a69-08a49a6465d2 nodeName:}" failed. No retries permitted until 2026-09-16 13:32:15.465585677 +0000 UTC m=+83.566398384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/557e9f3c-9f2c-45db-8a69-08a49a6465d2-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-q7vzb" (UID: "557e9f3c-9f2c-45db-8a69-08a49a6465d2") : secret "kube-state-metrics-tls" not found Sep 16 13:32:14.965846 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.965736 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/839cb0c4-5760-48c2-87ef-015f82193daf-node-exporter-textfile\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:14.965992 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.965911 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/557e9f3c-9f2c-45db-8a69-08a49a6465d2-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-q7vzb\" (UID: \"557e9f3c-9f2c-45db-8a69-08a49a6465d2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" Sep 16 13:32:14.965992 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.965915 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/557e9f3c-9f2c-45db-8a69-08a49a6465d2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-q7vzb\" (UID: \"557e9f3c-9f2c-45db-8a69-08a49a6465d2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" Sep 16 13:32:14.968055 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.968023 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/557e9f3c-9f2c-45db-8a69-08a49a6465d2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-q7vzb\" (UID: \"557e9f3c-9f2c-45db-8a69-08a49a6465d2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" Sep 16 13:32:14.976919 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:14.976898 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6jht\" (UniqueName: \"kubernetes.io/projected/557e9f3c-9f2c-45db-8a69-08a49a6465d2-kube-api-access-h6jht\") pod \"kube-state-metrics-59fbdd77d-q7vzb\" (UID: \"557e9f3c-9f2c-45db-8a69-08a49a6465d2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" Sep 16 13:32:15.066253 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.066171 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/839cb0c4-5760-48c2-87ef-015f82193daf-metrics-client-ca\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:15.066253 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.066205 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/839cb0c4-5760-48c2-87ef-015f82193daf-node-exporter-accelerators-collector-config\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:15.066253 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.066229 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/839cb0c4-5760-48c2-87ef-015f82193daf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:15.066253 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.066251 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/839cb0c4-5760-48c2-87ef-015f82193daf-sys\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:15.067223 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.066306 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/839cb0c4-5760-48c2-87ef-015f82193daf-node-exporter-wtmp\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:15.067223 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.066329 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfgw9\" (UniqueName: \"kubernetes.io/projected/839cb0c4-5760-48c2-87ef-015f82193daf-kube-api-access-wfgw9\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:15.067223 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.066355 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/839cb0c4-5760-48c2-87ef-015f82193daf-root\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:15.067223 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.066361 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/839cb0c4-5760-48c2-87ef-015f82193daf-sys\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:15.067223 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.066393 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/839cb0c4-5760-48c2-87ef-015f82193daf-node-exporter-textfile\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:15.067223 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.066423 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/839cb0c4-5760-48c2-87ef-015f82193daf-node-exporter-tls\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:15.067223 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:32:15.066520 2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 16 13:32:15.067223 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.066530 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/839cb0c4-5760-48c2-87ef-015f82193daf-root\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:15.067223 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:32:15.066581 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/839cb0c4-5760-48c2-87ef-015f82193daf-node-exporter-tls podName:839cb0c4-5760-48c2-87ef-015f82193daf nodeName:}" failed. No retries permitted until 2026-09-16 13:32:15.566564181 +0000 UTC m=+83.667376874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/839cb0c4-5760-48c2-87ef-015f82193daf-node-exporter-tls") pod "node-exporter-rwsmj" (UID: "839cb0c4-5760-48c2-87ef-015f82193daf") : secret "node-exporter-tls" not found Sep 16 13:32:15.067223 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.066690 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/839cb0c4-5760-48c2-87ef-015f82193daf-node-exporter-wtmp\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:15.067223 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.066804 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/839cb0c4-5760-48c2-87ef-015f82193daf-node-exporter-textfile\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:15.067223 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.067071 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/839cb0c4-5760-48c2-87ef-015f82193daf-node-exporter-accelerators-collector-config\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:15.067745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.067246 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/839cb0c4-5760-48c2-87ef-015f82193daf-metrics-client-ca\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:15.069706 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.069677 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/839cb0c4-5760-48c2-87ef-015f82193daf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:15.078338 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.078316 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfgw9\" (UniqueName: \"kubernetes.io/projected/839cb0c4-5760-48c2-87ef-015f82193daf-kube-api-access-wfgw9\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:15.106000 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.105969 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:15.204882 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.204851 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:15.307962 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.307936 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:15.404563 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.404486 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:15.469046 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.469001 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/557e9f3c-9f2c-45db-8a69-08a49a6465d2-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-q7vzb\" (UID: \"557e9f3c-9f2c-45db-8a69-08a49a6465d2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" Sep 16 13:32:15.471598 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.471569 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/557e9f3c-9f2c-45db-8a69-08a49a6465d2-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-q7vzb\" (UID: \"557e9f3c-9f2c-45db-8a69-08a49a6465d2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" Sep 16 13:32:15.474000 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.473970 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:15.569924 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.569891 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/839cb0c4-5760-48c2-87ef-015f82193daf-node-exporter-tls\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:15.572320 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.572293 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/839cb0c4-5760-48c2-87ef-015f82193daf-node-exporter-tls\") pod \"node-exporter-rwsmj\" (UID: \"839cb0c4-5760-48c2-87ef-015f82193daf\") " pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:15.723586 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.723556 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" Sep 16 13:32:15.748397 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.748366 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:15.764790 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.764716 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rwsmj" Sep 16 13:32:15.776780 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:32:15.775888 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839cb0c4_5760_48c2_87ef_015f82193daf.slice/crio-fc9133bf772a98807538fdd227c7d71f659f8f299576d8b54d44d2cee8d4d4ab WatchSource:0}: Error finding container fc9133bf772a98807538fdd227c7d71f659f8f299576d8b54d44d2cee8d4d4ab: Status 404 returned error can't find the container with id fc9133bf772a98807538fdd227c7d71f659f8f299576d8b54d44d2cee8d4d4ab Sep 16 13:32:15.864135 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.864068 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:15.911527 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.911495 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 13:32:15.914131 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.914115 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:15.928865 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:32:15.928828 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-kube-rbac-proxy-web\" is forbidden: User \"system:node:ip-10-0-132-9.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-132-9.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" type="*v1.Secret" Sep 16 13:32:15.928865 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:32:15.928841 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-dockercfg-82f57\" is forbidden: User \"system:node:ip-10-0-132-9.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-132-9.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-82f57\"" type="*v1.Secret" Sep 16 13:32:15.929058 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:32:15.928853 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-tls-assets-0\" is forbidden: User \"system:node:ip-10-0-132-9.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-132-9.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" type="*v1.Secret" Sep 16 13:32:15.929058 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:32:15.928855 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"alertmanager-main-0\" is forbidden: User \"system:node:ip-10-0-132-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-132-9.ec2.internal' and this object" podUID="804ff96f-49e3-4fd7-ade8-808e42744b89" pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:15.929058 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:32:15.928867 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-cluster-tls-config\" is forbidden: User \"system:node:ip-10-0-132-9.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-132-9.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" type="*v1.Secret" Sep 16 13:32:15.929058 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:32:15.928926 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-generated\" is forbidden: User \"system:node:ip-10-0-132-9.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-132-9.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" type="*v1.Secret" Sep 16 13:32:15.929058 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:32:15.928944 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-kube-rbac-proxy\" is forbidden: User \"system:node:ip-10-0-132-9.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-132-9.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" type="*v1.Secret" Sep 16 13:32:15.929058 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:32:15.928966 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-tls\" is forbidden: User \"system:node:ip-10-0-132-9.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-132-9.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" type="*v1.Secret" Sep 16 13:32:15.929058 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.928992 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 16 13:32:15.929058 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.929024 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 16 13:32:15.930195 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:32:15.930176 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-kube-rbac-proxy-metric\" is forbidden: User \"system:node:ip-10-0-132-9.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-132-9.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" type="*v1.Secret" Sep 16 13:32:15.972817 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.972791 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-web-config\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:15.972817 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.972824 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/804ff96f-49e3-4fd7-ade8-808e42744b89-config-out\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:15.973042 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.972843 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:15.973042 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.972873 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/804ff96f-49e3-4fd7-ade8-808e42744b89-tls-assets\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:15.973042 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.972962 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:15.973042 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.973002 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-config-volume\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:15.973042 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.973033 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cw2k\" (UniqueName: \"kubernetes.io/projected/804ff96f-49e3-4fd7-ade8-808e42744b89-kube-api-access-6cw2k\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:15.973302 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.973062 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/804ff96f-49e3-4fd7-ade8-808e42744b89-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:15.973302 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.973123 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:15.973302 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.973222 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:15.973458 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.973303 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:15.973458 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.973375 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/804ff96f-49e3-4fd7-ade8-808e42744b89-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:15.973458 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.973405 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/804ff96f-49e3-4fd7-ade8-808e42744b89-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:15.977510 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:15.977451 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 13:32:16.035679 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.035649 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb"] Sep 16 13:32:16.038801 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:32:16.038775 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557e9f3c_9f2c_45db_8a69_08a49a6465d2.slice/crio-2526749d8fa7ee3e3f7bd6285ffb6b33f47be8d6668afb34de65d5ed5e8956d4 WatchSource:0}: Error finding container 2526749d8fa7ee3e3f7bd6285ffb6b33f47be8d6668afb34de65d5ed5e8956d4: Status 404 returned error can't find the container with id 2526749d8fa7ee3e3f7bd6285ffb6b33f47be8d6668afb34de65d5ed5e8956d4 Sep 16 13:32:16.055035 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.055007 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:16.073827 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.073793 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-web-config\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:16.073914 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.073831 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/804ff96f-49e3-4fd7-ade8-808e42744b89-config-out\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:16.073914 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.073854 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:16.073914 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.073886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/804ff96f-49e3-4fd7-ade8-808e42744b89-tls-assets\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:16.074058 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.073930 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:16.074058 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.073959 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-config-volume\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:16.074058 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.073988 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cw2k\" (UniqueName: \"kubernetes.io/projected/804ff96f-49e3-4fd7-ade8-808e42744b89-kube-api-access-6cw2k\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:16.074201 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.074020 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/804ff96f-49e3-4fd7-ade8-808e42744b89-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:16.074201 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.074091 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:16.074201 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.074124 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:16.074201 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.074142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:16.074201 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:32:16.074168 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/804ff96f-49e3-4fd7-ade8-808e42744b89-alertmanager-trusted-ca-bundle podName:804ff96f-49e3-4fd7-ade8-808e42744b89 nodeName:}" failed. No retries permitted until 2026-09-16 13:32:16.574146283 +0000 UTC m=+84.674958994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/804ff96f-49e3-4fd7-ade8-808e42744b89-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "804ff96f-49e3-4fd7-ade8-808e42744b89") : configmap references non-existent config key: ca-bundle.crt Sep 16 13:32:16.074426 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.074207 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/804ff96f-49e3-4fd7-ade8-808e42744b89-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:16.074426 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.074239 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/804ff96f-49e3-4fd7-ade8-808e42744b89-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:16.074659 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.074643 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/804ff96f-49e3-4fd7-ade8-808e42744b89-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:16.074916 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.074900 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/804ff96f-49e3-4fd7-ade8-808e42744b89-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:16.076746 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.076730 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/804ff96f-49e3-4fd7-ade8-808e42744b89-config-out\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:16.076910 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.076893 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-web-config\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:16.083981 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.083960 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cw2k\" (UniqueName: \"kubernetes.io/projected/804ff96f-49e3-4fd7-ade8-808e42744b89-kube-api-access-6cw2k\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:16.189138 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.189110 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:16.440217 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.440136 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:16.510997 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.510962 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:16.578843 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.578811 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/804ff96f-49e3-4fd7-ade8-808e42744b89-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:16.579751 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.579718 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/804ff96f-49e3-4fd7-ade8-808e42744b89-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:16.712975 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.712891 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:16.773767 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.773732 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" event={"ID":"557e9f3c-9f2c-45db-8a69-08a49a6465d2","Type":"ContainerStarted","Data":"2526749d8fa7ee3e3f7bd6285ffb6b33f47be8d6668afb34de65d5ed5e8956d4"} Sep 16 13:32:16.775275 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.775240 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rwsmj" event={"ID":"839cb0c4-5760-48c2-87ef-015f82193daf","Type":"ContainerStarted","Data":"fc9133bf772a98807538fdd227c7d71f659f8f299576d8b54d44d2cee8d4d4ab"} Sep 16 13:32:16.807579 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.807539 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 16 13:32:16.810438 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.810390 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:16.817260 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:16.817234 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:17.014166 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.014132 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 16 13:32:17.016842 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.016815 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 16 13:32:17.017123 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.017102 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:17.026967 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.026936 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/804ff96f-49e3-4fd7-ade8-808e42744b89-tls-assets\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:17.072926 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.072893 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-82f57\"" Sep 16 13:32:17.074952 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:32:17.074931 2580 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: failed to sync secret cache: timed out waiting for the condition Sep 16 13:32:17.074952 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:32:17.074941 2580 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-generated: failed to sync secret cache: timed out waiting for the condition Sep 16 13:32:17.075115 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:32:17.074950 2580 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-kube-rbac-proxy-web: failed to sync secret cache: timed out waiting for the condition Sep 16 13:32:17.075115 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:32:17.074970 2580 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-kube-rbac-proxy: failed to sync secret cache: timed out waiting for the condition Sep 16 13:32:17.075115 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:32:17.075005 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-secret-alertmanager-main-tls podName:804ff96f-49e3-4fd7-ade8-808e42744b89 nodeName:}" failed. No retries permitted until 2026-09-16 13:32:17.574986809 +0000 UTC m=+85.675799515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "804ff96f-49e3-4fd7-ade8-808e42744b89") : failed to sync secret cache: timed out waiting for the condition Sep 16 13:32:17.075115 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:32:17.075020 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-config-volume podName:804ff96f-49e3-4fd7-ade8-808e42744b89 nodeName:}" failed. No retries permitted until 2026-09-16 13:32:17.575014336 +0000 UTC m=+85.675827029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "config-volume" (UniqueName: "kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-config-volume") pod "alertmanager-main-0" (UID: "804ff96f-49e3-4fd7-ade8-808e42744b89") : failed to sync secret cache: timed out waiting for the condition Sep 16 13:32:17.075115 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:32:17.075043 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-secret-alertmanager-kube-rbac-proxy podName:804ff96f-49e3-4fd7-ade8-808e42744b89 nodeName:}" failed. No retries permitted until 2026-09-16 13:32:17.575025318 +0000 UTC m=+85.675838029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-kube-rbac-proxy" (UniqueName: "kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-secret-alertmanager-kube-rbac-proxy") pod "alertmanager-main-0" (UID: "804ff96f-49e3-4fd7-ade8-808e42744b89") : failed to sync secret cache: timed out waiting for the condition Sep 16 13:32:17.075115 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:32:17.075063 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-secret-alertmanager-kube-rbac-proxy-web podName:804ff96f-49e3-4fd7-ade8-808e42744b89 nodeName:}" failed. No retries permitted until 2026-09-16 13:32:17.57505405 +0000 UTC m=+85.675866749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-kube-rbac-proxy-web" (UniqueName: "kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-secret-alertmanager-kube-rbac-proxy-web") pod "alertmanager-main-0" (UID: "804ff96f-49e3-4fd7-ade8-808e42744b89") : failed to sync secret cache: timed out waiting for the condition Sep 16 13:32:17.135970 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.135934 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 16 13:32:17.193272 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.193235 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 16 13:32:17.334538 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.334459 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 16 13:32:17.427409 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.427382 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 16 13:32:17.586708 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.586617 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:17.586708 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.586667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-config-volume\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:17.587199 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.586720 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:17.587199 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.586792 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:17.589326 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.589053 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:17.589437 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.589326 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:17.589594 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.589575 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-config-volume\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:17.589661 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.589632 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/804ff96f-49e3-4fd7-ade8-808e42744b89-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"804ff96f-49e3-4fd7-ade8-808e42744b89\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:17.722913 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.722877 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 13:32:17.768693 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.768662 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:17.784671 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.784627 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" event={"ID":"557e9f3c-9f2c-45db-8a69-08a49a6465d2","Type":"ContainerStarted","Data":"67000ff29cf39e9836fdb4604bd976437ff3427ab56d6927880a19f6d014e101"} Sep 16 13:32:17.784811 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.784730 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" event={"ID":"557e9f3c-9f2c-45db-8a69-08a49a6465d2","Type":"ContainerStarted","Data":"c6179318afda2004fd9b792ed8b850b231e0df21ff9f107b5996cea69c97aea9"} Sep 16 13:32:17.784811 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.784759 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" event={"ID":"557e9f3c-9f2c-45db-8a69-08a49a6465d2","Type":"ContainerStarted","Data":"1840c352ce0f75f68ea9ecd7cfd8a0bd754dd1c74600355714ae15f7095fbaf6"} Sep 16 13:32:17.786813 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.786789 2580 generic.go:356] "Generic (PLEG): container finished" podID="839cb0c4-5760-48c2-87ef-015f82193daf" containerID="882ca71c8f141641b7cf1c86d5f493a732a69d890a9d9e78a3109a10404348ad" exitCode=0 Sep 16 13:32:17.786944 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.786841 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rwsmj" event={"ID":"839cb0c4-5760-48c2-87ef-015f82193daf","Type":"ContainerDied","Data":"882ca71c8f141641b7cf1c86d5f493a732a69d890a9d9e78a3109a10404348ad"} Sep 16 13:32:17.807677 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.807579 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q7vzb" podStartSLOduration=2.462224453 podStartE2EDuration="3.807562569s" totalImagesPullingTime="1.345338116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:32:17.806434339 +0000 UTC m=+85.907247055" watchObservedRunningTime="2026-09-16 13:32:17.807562569 +0000 UTC m=+85.908375283" Sep 16 13:32:17.836776 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:17.836713 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:18.008407 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:18.008347 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 13:32:18.011104 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:32:18.011062 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod804ff96f_49e3_4fd7_ade8_808e42744b89.slice/crio-2f228c27f5a5c36fe36a26bf7916b10afdbc985b286da37f7754c8d1d5ea02e1 WatchSource:0}: Error finding container 2f228c27f5a5c36fe36a26bf7916b10afdbc985b286da37f7754c8d1d5ea02e1: Status 404 returned error can't find the container with id 2f228c27f5a5c36fe36a26bf7916b10afdbc985b286da37f7754c8d1d5ea02e1 Sep 16 13:32:18.087263 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:18.087162 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:18.185013 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:18.184970 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:18.384109 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:18.384004 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:18.480253 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:18.480220 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:18.791897 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:18.791857 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"804ff96f-49e3-4fd7-ade8-808e42744b89","Type":"ContainerStarted","Data":"2f228c27f5a5c36fe36a26bf7916b10afdbc985b286da37f7754c8d1d5ea02e1"} Sep 16 13:32:18.793973 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:18.793947 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rwsmj" event={"ID":"839cb0c4-5760-48c2-87ef-015f82193daf","Type":"ContainerStarted","Data":"9094ef89e36193de7e47eeac28ea64cffbc461033f5326776cdbde7fbe03ecc6"} Sep 16 13:32:18.794085 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:18.793992 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rwsmj" event={"ID":"839cb0c4-5760-48c2-87ef-015f82193daf","Type":"ContainerStarted","Data":"8ba01b6657fcf5f9d503433b507b2397dfc29033f6259e46b2beee9c962a0082"} Sep 16 13:32:18.820037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:18.819989 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rwsmj" podStartSLOduration=3.6494132500000003 podStartE2EDuration="4.819973531s" totalImagesPullingTime="1.090746161s" totalInitContainerRuntime="79.81412ms" isStatefulPod=true podCreationTimestamp="2026-09-16 13:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:32:18.819251656 +0000 UTC m=+86.920064387" watchObservedRunningTime="2026-09-16 13:32:18.819973531 +0000 UTC m=+86.920786227" Sep 16 13:32:19.559572 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.559536 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:19.577244 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.577212 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fb6885f6f-7g7vp"] Sep 16 13:32:19.580154 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.580138 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:19.584389 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.584368 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 16 13:32:19.584672 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.584653 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 16 13:32:19.585635 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.585623 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 16 13:32:19.585887 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.585867 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 16 13:32:19.585939 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.585902 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 16 13:32:19.585939 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.585927 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 16 13:32:19.586033 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.585997 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 16 13:32:19.586167 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.586154 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-j4xqz\"" Sep 16 13:32:19.594034 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.594011 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 16 13:32:19.595934 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.595911 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fb6885f6f-7g7vp"] Sep 16 13:32:19.603662 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.603538 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-oauth-serving-cert\") pod \"console-6fb6885f6f-7g7vp\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:19.603662 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.603592 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11a17f72-796a-46e0-b646-9a194c8b4795-console-oauth-config\") pod \"console-6fb6885f6f-7g7vp\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:19.603662 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.603633 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11a17f72-796a-46e0-b646-9a194c8b4795-console-serving-cert\") pod \"console-6fb6885f6f-7g7vp\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:19.603880 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.603665 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfdqn\" (UniqueName: \"kubernetes.io/projected/11a17f72-796a-46e0-b646-9a194c8b4795-kube-api-access-jfdqn\") pod \"console-6fb6885f6f-7g7vp\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:19.603880 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.603714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-console-config\") pod \"console-6fb6885f6f-7g7vp\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:19.603880 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.603783 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-service-ca\") pod \"console-6fb6885f6f-7g7vp\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:19.603880 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.603822 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-trusted-ca-bundle\") pod \"console-6fb6885f6f-7g7vp\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:19.651720 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.651688 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:19.704647 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.704550 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfdqn\" (UniqueName: \"kubernetes.io/projected/11a17f72-796a-46e0-b646-9a194c8b4795-kube-api-access-jfdqn\") pod \"console-6fb6885f6f-7g7vp\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:19.704647 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.704594 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-console-config\") pod \"console-6fb6885f6f-7g7vp\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:19.704647 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.704640 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-service-ca\") pod \"console-6fb6885f6f-7g7vp\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:19.704879 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.704673 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-trusted-ca-bundle\") pod \"console-6fb6885f6f-7g7vp\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:19.704879 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.704694 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-oauth-serving-cert\") pod \"console-6fb6885f6f-7g7vp\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:19.704879 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.704715 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11a17f72-796a-46e0-b646-9a194c8b4795-console-oauth-config\") pod \"console-6fb6885f6f-7g7vp\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:19.704879 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.704730 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11a17f72-796a-46e0-b646-9a194c8b4795-console-serving-cert\") pod \"console-6fb6885f6f-7g7vp\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:19.705450 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.705423 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-console-config\") pod \"console-6fb6885f6f-7g7vp\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:19.705737 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.705423 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-service-ca\") pod \"console-6fb6885f6f-7g7vp\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:19.705737 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.705512 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-oauth-serving-cert\") pod \"console-6fb6885f6f-7g7vp\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:19.705737 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.705581 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-trusted-ca-bundle\") pod \"console-6fb6885f6f-7g7vp\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:19.707293 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.707275 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11a17f72-796a-46e0-b646-9a194c8b4795-console-serving-cert\") pod \"console-6fb6885f6f-7g7vp\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:19.707374 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.707324 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11a17f72-796a-46e0-b646-9a194c8b4795-console-oauth-config\") pod \"console-6fb6885f6f-7g7vp\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:19.714468 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.714448 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfdqn\" (UniqueName: \"kubernetes.io/projected/11a17f72-796a-46e0-b646-9a194c8b4795-kube-api-access-jfdqn\") pod \"console-6fb6885f6f-7g7vp\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:19.798822 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.798783 2580 generic.go:356] "Generic (PLEG): container finished" podID="804ff96f-49e3-4fd7-ade8-808e42744b89" containerID="80c6162a680372fdcd5f670e60998e2aa5e5ec44f306224e9afd172941ffa1a4" exitCode=0 Sep 16 13:32:19.799269 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.798873 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"804ff96f-49e3-4fd7-ade8-808e42744b89","Type":"ContainerDied","Data":"80c6162a680372fdcd5f670e60998e2aa5e5ec44f306224e9afd172941ffa1a4"} Sep 16 13:32:19.876759 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.876722 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:19.898526 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:19.898487 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:20.142416 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:20.142381 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:20.173673 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:20.173644 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fb6885f6f-7g7vp"] Sep 16 13:32:20.176830 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:32:20.176803 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a17f72_796a_46e0_b646_9a194c8b4795.slice/crio-e9a320e2168578a3e0357e1ec88f40d225045d6e5e2a81fcb6f2f1a604050153 WatchSource:0}: Error finding container e9a320e2168578a3e0357e1ec88f40d225045d6e5e2a81fcb6f2f1a604050153: Status 404 returned error can't find the container with id e9a320e2168578a3e0357e1ec88f40d225045d6e5e2a81fcb6f2f1a604050153 Sep 16 13:32:20.243717 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:20.243682 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:20.535316 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:20.535260 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:20.780042 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:20.780004 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:20.804600 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:20.804513 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fb6885f6f-7g7vp" event={"ID":"11a17f72-796a-46e0-b646-9a194c8b4795","Type":"ContainerStarted","Data":"e9a320e2168578a3e0357e1ec88f40d225045d6e5e2a81fcb6f2f1a604050153"} Sep 16 13:32:21.261505 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:21.261477 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:21.462355 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:21.462320 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:21.811203 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:21.811172 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"804ff96f-49e3-4fd7-ade8-808e42744b89","Type":"ContainerStarted","Data":"bf54ac5e512da91a7e17c349a283f76bd4c239f3c38d0b009b66342bb9174b55"} Sep 16 13:32:22.819008 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:22.818969 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"804ff96f-49e3-4fd7-ade8-808e42744b89","Type":"ContainerStarted","Data":"4856460c1d68dfaa4dbaed64b0c234b55edf75c6ddd3904f760fae616fd20ef7"} Sep 16 13:32:22.819400 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:22.819046 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"804ff96f-49e3-4fd7-ade8-808e42744b89","Type":"ContainerStarted","Data":"37a7513e3de8cf8d175e80b92a812e7c69d5bcb9ce2aa46c9f3c15817ff89fa0"} Sep 16 13:32:22.819400 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:22.819072 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"804ff96f-49e3-4fd7-ade8-808e42744b89","Type":"ContainerStarted","Data":"50f080ac5e366ffca80bb4cca9a4d79817349f62b470fe5131b638b729579087"} Sep 16 13:32:22.819400 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:22.819099 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"804ff96f-49e3-4fd7-ade8-808e42744b89","Type":"ContainerStarted","Data":"3831faddcc3e74c825b2bdac96e407091133b38f7d99716e21b4be9cd1c97287"} Sep 16 13:32:23.825183 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:23.825131 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fb6885f6f-7g7vp" event={"ID":"11a17f72-796a-46e0-b646-9a194c8b4795","Type":"ContainerStarted","Data":"e63aa8476cad1566ceec17d0f746f1f76cbdb07d782e25b959c4ffdbff20b47a"} Sep 16 13:32:23.849313 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:23.849251 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6fb6885f6f-7g7vp" podStartSLOduration=1.39875951 podStartE2EDuration="4.849229198s" totalImagesPullingTime="3.450469688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:32:23.848129183 +0000 UTC m=+91.948941902" watchObservedRunningTime="2026-09-16 13:32:23.849229198 +0000 UTC m=+91.950041913" Sep 16 13:32:24.831920 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:24.831879 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"804ff96f-49e3-4fd7-ade8-808e42744b89","Type":"ContainerStarted","Data":"730beb6a76fd164087cc26b03483aff81e12064c5af3c1a727a1f23ae70f28a9"} Sep 16 13:32:24.866022 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:24.865972 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.603245923 podStartE2EDuration="9.865956125s" totalImagesPullingTime="5.246271158s" totalInitContainerRuntime="16.439044ms" isStatefulPod=false podCreationTimestamp="2026-09-16 13:32:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:32:24.865279327 +0000 UTC m=+92.966092070" watchObservedRunningTime="2026-09-16 13:32:24.865956125 +0000 UTC m=+92.966768840" Sep 16 13:32:25.154397 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:25.154318 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:25.256089 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:25.256056 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:29.899427 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:29.899394 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:29.899830 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:29.899473 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:29.904405 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:29.904380 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:30.857399 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:30.857367 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:32:31.231620 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:31.231587 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:31.324148 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:31.324112 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:33.278192 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:33.278155 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:33.402353 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:33.402320 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:33.867204 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:33.867172 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:35.735841 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:35.735805 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m6j5f" Sep 16 13:32:48.306209 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:48.306173 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:48.371495 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:48.371426 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:52.301789 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:32:52.301756 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:33:34.140048 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:34.140012 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:33:34.411420 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:34.411339 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:33:39.236625 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.236572 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk"] Sep 16 13:33:39.241363 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.241343 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.244159 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.244135 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 16 13:33:39.244277 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.244155 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 16 13:33:39.244277 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.244139 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 16 13:33:39.244277 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.244197 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 16 13:33:39.244277 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.244210 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fx6vh\"" Sep 16 13:33:39.244520 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.244503 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 16 13:33:39.250737 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.250722 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 16 13:33:39.253197 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.253178 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk"] Sep 16 13:33:39.288654 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.288598 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da9faeb3-35c3-484c-80a6-7da2accdb5ab-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.288906 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.288672 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da9faeb3-35c3-484c-80a6-7da2accdb5ab-serving-certs-ca-bundle\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.288906 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.288730 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgpjz\" (UniqueName: \"kubernetes.io/projected/da9faeb3-35c3-484c-80a6-7da2accdb5ab-kube-api-access-zgpjz\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.288906 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.288768 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/da9faeb3-35c3-484c-80a6-7da2accdb5ab-telemeter-client-tls\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.288906 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.288807 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da9faeb3-35c3-484c-80a6-7da2accdb5ab-metrics-client-ca\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.288906 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.288843 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/da9faeb3-35c3-484c-80a6-7da2accdb5ab-federate-client-tls\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.288906 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.288866 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da9faeb3-35c3-484c-80a6-7da2accdb5ab-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.288906 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.288903 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/da9faeb3-35c3-484c-80a6-7da2accdb5ab-secret-telemeter-client\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.389384 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.389332 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da9faeb3-35c3-484c-80a6-7da2accdb5ab-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.389635 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.389498 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da9faeb3-35c3-484c-80a6-7da2accdb5ab-serving-certs-ca-bundle\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.389635 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.389518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgpjz\" (UniqueName: \"kubernetes.io/projected/da9faeb3-35c3-484c-80a6-7da2accdb5ab-kube-api-access-zgpjz\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.389635 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.389535 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/da9faeb3-35c3-484c-80a6-7da2accdb5ab-telemeter-client-tls\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.389826 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.389775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da9faeb3-35c3-484c-80a6-7da2accdb5ab-metrics-client-ca\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.389881 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.389868 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/da9faeb3-35c3-484c-80a6-7da2accdb5ab-federate-client-tls\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.389934 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.389897 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da9faeb3-35c3-484c-80a6-7da2accdb5ab-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.390009 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.389986 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/da9faeb3-35c3-484c-80a6-7da2accdb5ab-secret-telemeter-client\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.390313 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.390284 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da9faeb3-35c3-484c-80a6-7da2accdb5ab-serving-certs-ca-bundle\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.390419 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.390347 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da9faeb3-35c3-484c-80a6-7da2accdb5ab-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.390900 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.390873 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da9faeb3-35c3-484c-80a6-7da2accdb5ab-metrics-client-ca\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.392511 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.392481 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/da9faeb3-35c3-484c-80a6-7da2accdb5ab-telemeter-client-tls\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.392661 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.392640 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/da9faeb3-35c3-484c-80a6-7da2accdb5ab-secret-telemeter-client\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.392771 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.392755 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da9faeb3-35c3-484c-80a6-7da2accdb5ab-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.392809 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.392787 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/da9faeb3-35c3-484c-80a6-7da2accdb5ab-federate-client-tls\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.398806 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.398782 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgpjz\" (UniqueName: \"kubernetes.io/projected/da9faeb3-35c3-484c-80a6-7da2accdb5ab-kube-api-access-zgpjz\") pod \"telemeter-client-68fb5d7d69-ldgsk\" (UID: \"da9faeb3-35c3-484c-80a6-7da2accdb5ab\") " pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.551089 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.550996 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" Sep 16 13:33:39.830638 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:39.830583 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk"] Sep 16 13:33:39.831980 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:33:39.831953 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9faeb3_35c3_484c_80a6_7da2accdb5ab.slice/crio-8bb1da10d33939c6f92fcd293f471d289f2503fe0d07e36bc1fa3f9e16818e49 WatchSource:0}: Error finding container 8bb1da10d33939c6f92fcd293f471d289f2503fe0d07e36bc1fa3f9e16818e49: Status 404 returned error can't find the container with id 8bb1da10d33939c6f92fcd293f471d289f2503fe0d07e36bc1fa3f9e16818e49 Sep 16 13:33:40.095822 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:40.095734 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" event={"ID":"da9faeb3-35c3-484c-80a6-7da2accdb5ab","Type":"ContainerStarted","Data":"8bb1da10d33939c6f92fcd293f471d289f2503fe0d07e36bc1fa3f9e16818e49"} Sep 16 13:33:42.105881 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.105844 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" event={"ID":"da9faeb3-35c3-484c-80a6-7da2accdb5ab","Type":"ContainerStarted","Data":"cb5ed279857bcaa3a67b050db2f4def62bdf6bf51010497176fb8b53a6d5f159"} Sep 16 13:33:42.106356 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.105900 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" event={"ID":"da9faeb3-35c3-484c-80a6-7da2accdb5ab","Type":"ContainerStarted","Data":"e2490fc89501f434935e53bad8bfa7f0679b949dd1841f2dbc885880674f1437"} Sep 16 13:33:42.106356 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.105921 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" event={"ID":"da9faeb3-35c3-484c-80a6-7da2accdb5ab","Type":"ContainerStarted","Data":"7233ebbcaaf402ce8feebc165ff11c5b34e9c7084c93c16c34df5752d3629570"} Sep 16 13:33:42.131545 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.131493 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-68fb5d7d69-ldgsk" podStartSLOduration=1.424343498 podStartE2EDuration="3.13147548s" totalImagesPullingTime="1.707131982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:33:42.129563936 +0000 UTC m=+170.230376837" watchObservedRunningTime="2026-09-16 13:33:42.13147548 +0000 UTC m=+170.232288195" Sep 16 13:33:42.228232 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.228193 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f58bd7fff-lzprj"] Sep 16 13:33:42.231345 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.231323 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.245901 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.245875 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f58bd7fff-lzprj"] Sep 16 13:33:42.315187 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.315140 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-oauth-serving-cert\") pod \"console-f58bd7fff-lzprj\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.315187 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.315189 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac39ec5a-b502-4320-9d1d-dc926718c2b2-console-serving-cert\") pod \"console-f58bd7fff-lzprj\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.315438 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.315210 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-console-config\") pod \"console-f58bd7fff-lzprj\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.315438 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.315269 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-service-ca\") pod \"console-f58bd7fff-lzprj\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.315438 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.315338 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btbfk\" (UniqueName: \"kubernetes.io/projected/ac39ec5a-b502-4320-9d1d-dc926718c2b2-kube-api-access-btbfk\") pod \"console-f58bd7fff-lzprj\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.315438 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.315364 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-trusted-ca-bundle\") pod \"console-f58bd7fff-lzprj\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.315438 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.315404 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac39ec5a-b502-4320-9d1d-dc926718c2b2-console-oauth-config\") pod \"console-f58bd7fff-lzprj\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.316223 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.316200 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:33:42.415951 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.415855 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac39ec5a-b502-4320-9d1d-dc926718c2b2-console-serving-cert\") pod \"console-f58bd7fff-lzprj\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.415951 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.415913 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-console-config\") pod \"console-f58bd7fff-lzprj\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.415951 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.415937 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-service-ca\") pod \"console-f58bd7fff-lzprj\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.416200 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.415988 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btbfk\" (UniqueName: \"kubernetes.io/projected/ac39ec5a-b502-4320-9d1d-dc926718c2b2-kube-api-access-btbfk\") pod \"console-f58bd7fff-lzprj\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.416200 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.416017 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-trusted-ca-bundle\") pod \"console-f58bd7fff-lzprj\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.416200 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.416168 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac39ec5a-b502-4320-9d1d-dc926718c2b2-console-oauth-config\") pod \"console-f58bd7fff-lzprj\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.416343 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.416237 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-oauth-serving-cert\") pod \"console-f58bd7fff-lzprj\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.416343 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.416277 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:33:42.416954 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.416928 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-service-ca\") pod \"console-f58bd7fff-lzprj\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.417103 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.417081 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-console-config\") pod \"console-f58bd7fff-lzprj\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.417169 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.417093 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-trusted-ca-bundle\") pod \"console-f58bd7fff-lzprj\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.417227 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.417197 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-oauth-serving-cert\") pod \"console-f58bd7fff-lzprj\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.418655 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.418633 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac39ec5a-b502-4320-9d1d-dc926718c2b2-console-serving-cert\") pod \"console-f58bd7fff-lzprj\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.418772 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.418753 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac39ec5a-b502-4320-9d1d-dc926718c2b2-console-oauth-config\") pod \"console-f58bd7fff-lzprj\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.424552 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.424529 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btbfk\" (UniqueName: \"kubernetes.io/projected/ac39ec5a-b502-4320-9d1d-dc926718c2b2-kube-api-access-btbfk\") pod \"console-f58bd7fff-lzprj\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.540121 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.540081 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:42.814884 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:42.814864 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f58bd7fff-lzprj"] Sep 16 13:33:42.816869 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:33:42.816837 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac39ec5a_b502_4320_9d1d_dc926718c2b2.slice/crio-9817da109b0b5160eba249c6f04ab8c8b33bdd72d2cfe55335ed004fe9f5bce4 WatchSource:0}: Error finding container 9817da109b0b5160eba249c6f04ab8c8b33bdd72d2cfe55335ed004fe9f5bce4: Status 404 returned error can't find the container with id 9817da109b0b5160eba249c6f04ab8c8b33bdd72d2cfe55335ed004fe9f5bce4 Sep 16 13:33:43.111806 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:43.111701 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f58bd7fff-lzprj" event={"ID":"ac39ec5a-b502-4320-9d1d-dc926718c2b2","Type":"ContainerStarted","Data":"17b6f615a3b3bcc4841713096b843cb28b55d517000ef873d2acfb5a71060960"} Sep 16 13:33:43.111806 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:43.111756 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f58bd7fff-lzprj" event={"ID":"ac39ec5a-b502-4320-9d1d-dc926718c2b2","Type":"ContainerStarted","Data":"9817da109b0b5160eba249c6f04ab8c8b33bdd72d2cfe55335ed004fe9f5bce4"} Sep 16 13:33:43.133466 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:43.133410 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f58bd7fff-lzprj" podStartSLOduration=1.133393697 podStartE2EDuration="1.133393697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:33:43.132268718 +0000 UTC m=+171.233081434" watchObservedRunningTime="2026-09-16 13:33:43.133393697 +0000 UTC m=+171.234206413" Sep 16 13:33:49.515416 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.515375 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f58bd7fff-lzprj"] Sep 16 13:33:49.527508 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.527476 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:33:49.552961 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.552930 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dc777cfcf-vxwsq"] Sep 16 13:33:49.556194 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.556179 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:49.566223 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.566201 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dc777cfcf-vxwsq"] Sep 16 13:33:49.608663 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.608625 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:33:49.671168 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.671137 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-console-oauth-config\") pod \"console-7dc777cfcf-vxwsq\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:49.671290 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.671178 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-oauth-serving-cert\") pod \"console-7dc777cfcf-vxwsq\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:49.671290 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.671197 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-console-serving-cert\") pod \"console-7dc777cfcf-vxwsq\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:49.671290 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.671253 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-console-config\") pod \"console-7dc777cfcf-vxwsq\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:49.671397 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.671345 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-trusted-ca-bundle\") pod \"console-7dc777cfcf-vxwsq\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:49.671397 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.671371 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c92j8\" (UniqueName: \"kubernetes.io/projected/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-kube-api-access-c92j8\") pod \"console-7dc777cfcf-vxwsq\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:49.671504 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.671396 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-service-ca\") pod \"console-7dc777cfcf-vxwsq\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:49.771981 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.771897 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-trusted-ca-bundle\") pod \"console-7dc777cfcf-vxwsq\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:49.771981 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.771937 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c92j8\" (UniqueName: \"kubernetes.io/projected/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-kube-api-access-c92j8\") pod \"console-7dc777cfcf-vxwsq\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:49.771981 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.771960 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-service-ca\") pod \"console-7dc777cfcf-vxwsq\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:49.772226 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.772003 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-console-oauth-config\") pod \"console-7dc777cfcf-vxwsq\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:49.772226 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.772041 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-oauth-serving-cert\") pod \"console-7dc777cfcf-vxwsq\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:49.772226 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.772067 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-console-serving-cert\") pod \"console-7dc777cfcf-vxwsq\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:49.772226 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.772093 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-console-config\") pod \"console-7dc777cfcf-vxwsq\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:49.772862 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.772832 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-service-ca\") pod \"console-7dc777cfcf-vxwsq\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:49.772999 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.772897 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-trusted-ca-bundle\") pod \"console-7dc777cfcf-vxwsq\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:49.772999 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.772865 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-oauth-serving-cert\") pod \"console-7dc777cfcf-vxwsq\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:49.773082 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.772967 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-console-config\") pod \"console-7dc777cfcf-vxwsq\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:49.774671 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.774646 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-console-oauth-config\") pod \"console-7dc777cfcf-vxwsq\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:49.774817 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.774798 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-console-serving-cert\") pod \"console-7dc777cfcf-vxwsq\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:49.781840 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.781824 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c92j8\" (UniqueName: \"kubernetes.io/projected/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-kube-api-access-c92j8\") pod \"console-7dc777cfcf-vxwsq\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:49.864938 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:49.864901 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:50.140835 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:50.140812 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dc777cfcf-vxwsq"] Sep 16 13:33:50.143261 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:33:50.143240 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d8caf0_7b53_459c_a11b_d4a56bc28b7b.slice/crio-2e01d668894dee2e282b6d53de5b13e66fca2f0d3f8e8ffe032dd486c6200836 WatchSource:0}: Error finding container 2e01d668894dee2e282b6d53de5b13e66fca2f0d3f8e8ffe032dd486c6200836: Status 404 returned error can't find the container with id 2e01d668894dee2e282b6d53de5b13e66fca2f0d3f8e8ffe032dd486c6200836 Sep 16 13:33:51.142251 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:51.142218 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc777cfcf-vxwsq" event={"ID":"19d8caf0-7b53-459c-a11b-d4a56bc28b7b","Type":"ContainerStarted","Data":"09cd445fca4e056aab3e1840767ca053316efebb9ad5c168b2eb994b7fdca8bc"} Sep 16 13:33:51.142655 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:51.142267 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc777cfcf-vxwsq" event={"ID":"19d8caf0-7b53-459c-a11b-d4a56bc28b7b","Type":"ContainerStarted","Data":"2e01d668894dee2e282b6d53de5b13e66fca2f0d3f8e8ffe032dd486c6200836"} Sep 16 13:33:51.170260 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:51.170205 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7dc777cfcf-vxwsq" podStartSLOduration=2.170187561 podStartE2EDuration="2.170187561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:33:51.167940777 +0000 UTC m=+179.268753493" watchObservedRunningTime="2026-09-16 13:33:51.170187561 +0000 UTC m=+179.271000276" Sep 16 13:33:52.541035 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:52.540992 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:33:59.865814 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:59.865770 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:59.866356 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:59.865832 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:33:59.870725 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:33:59.870701 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:34:00.179850 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:00.179766 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:34:00.235395 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:00.235367 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fb6885f6f-7g7vp"] Sep 16 13:34:00.888578 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:00.888528 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:34:00.959695 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:00.959665 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:34:01.443734 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:01.443676 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:34:01.509541 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:01.509511 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:34:14.534884 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.534817 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f58bd7fff-lzprj" podUID="ac39ec5a-b502-4320-9d1d-dc926718c2b2" containerName="console" containerID="cri-o://17b6f615a3b3bcc4841713096b843cb28b55d517000ef873d2acfb5a71060960" gracePeriod=15 Sep 16 13:34:14.770883 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.770855 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:34:14.883585 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.883495 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-oauth-serving-cert\") pod \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " Sep 16 13:34:14.883585 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.883550 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-trusted-ca-bundle\") pod \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " Sep 16 13:34:14.883585 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.883573 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac39ec5a-b502-4320-9d1d-dc926718c2b2-console-serving-cert\") pod \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " Sep 16 13:34:14.883852 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.883637 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btbfk\" (UniqueName: \"kubernetes.io/projected/ac39ec5a-b502-4320-9d1d-dc926718c2b2-kube-api-access-btbfk\") pod \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " Sep 16 13:34:14.883852 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.883659 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-console-config\") pod \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " Sep 16 13:34:14.883852 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.883696 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac39ec5a-b502-4320-9d1d-dc926718c2b2-console-oauth-config\") pod \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " Sep 16 13:34:14.883852 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.883765 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-service-ca\") pod \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\" (UID: \"ac39ec5a-b502-4320-9d1d-dc926718c2b2\") " Sep 16 13:34:14.884043 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.883932 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ac39ec5a-b502-4320-9d1d-dc926718c2b2" (UID: "ac39ec5a-b502-4320-9d1d-dc926718c2b2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:34:14.884128 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.884095 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ac39ec5a-b502-4320-9d1d-dc926718c2b2" (UID: "ac39ec5a-b502-4320-9d1d-dc926718c2b2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:34:14.884256 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.884166 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-console-config" (OuterVolumeSpecName: "console-config") pod "ac39ec5a-b502-4320-9d1d-dc926718c2b2" (UID: "ac39ec5a-b502-4320-9d1d-dc926718c2b2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:34:14.884256 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.884211 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-service-ca" (OuterVolumeSpecName: "service-ca") pod "ac39ec5a-b502-4320-9d1d-dc926718c2b2" (UID: "ac39ec5a-b502-4320-9d1d-dc926718c2b2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:34:14.886167 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.886144 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac39ec5a-b502-4320-9d1d-dc926718c2b2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ac39ec5a-b502-4320-9d1d-dc926718c2b2" (UID: "ac39ec5a-b502-4320-9d1d-dc926718c2b2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:34:14.886271 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.886188 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac39ec5a-b502-4320-9d1d-dc926718c2b2-kube-api-access-btbfk" (OuterVolumeSpecName: "kube-api-access-btbfk") pod "ac39ec5a-b502-4320-9d1d-dc926718c2b2" (UID: "ac39ec5a-b502-4320-9d1d-dc926718c2b2"). InnerVolumeSpecName "kube-api-access-btbfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:34:14.886271 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.886190 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac39ec5a-b502-4320-9d1d-dc926718c2b2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ac39ec5a-b502-4320-9d1d-dc926718c2b2" (UID: "ac39ec5a-b502-4320-9d1d-dc926718c2b2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:34:14.984795 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.984748 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-service-ca\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:34:14.984795 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.984791 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-oauth-serving-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:34:14.984795 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.984801 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-trusted-ca-bundle\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:34:14.984795 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.984809 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac39ec5a-b502-4320-9d1d-dc926718c2b2-console-serving-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:34:14.985061 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.984817 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btbfk\" (UniqueName: \"kubernetes.io/projected/ac39ec5a-b502-4320-9d1d-dc926718c2b2-kube-api-access-btbfk\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:34:14.985061 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.984825 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac39ec5a-b502-4320-9d1d-dc926718c2b2-console-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:34:14.985061 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:14.984833 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac39ec5a-b502-4320-9d1d-dc926718c2b2-console-oauth-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:34:15.235980 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:15.235945 2580 generic.go:356] "Generic (PLEG): container finished" podID="ac39ec5a-b502-4320-9d1d-dc926718c2b2" containerID="17b6f615a3b3bcc4841713096b843cb28b55d517000ef873d2acfb5a71060960" exitCode=0 Sep 16 13:34:15.236185 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:15.235987 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f58bd7fff-lzprj" event={"ID":"ac39ec5a-b502-4320-9d1d-dc926718c2b2","Type":"ContainerDied","Data":"17b6f615a3b3bcc4841713096b843cb28b55d517000ef873d2acfb5a71060960"} Sep 16 13:34:15.236185 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:15.235999 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f58bd7fff-lzprj" Sep 16 13:34:15.236185 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:15.236027 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f58bd7fff-lzprj" event={"ID":"ac39ec5a-b502-4320-9d1d-dc926718c2b2","Type":"ContainerDied","Data":"9817da109b0b5160eba249c6f04ab8c8b33bdd72d2cfe55335ed004fe9f5bce4"} Sep 16 13:34:15.236185 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:15.236060 2580 scope.go:117] "RemoveContainer" containerID="17b6f615a3b3bcc4841713096b843cb28b55d517000ef873d2acfb5a71060960" Sep 16 13:34:15.244665 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:15.244647 2580 scope.go:117] "RemoveContainer" containerID="17b6f615a3b3bcc4841713096b843cb28b55d517000ef873d2acfb5a71060960" Sep 16 13:34:15.244961 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:34:15.244933 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17b6f615a3b3bcc4841713096b843cb28b55d517000ef873d2acfb5a71060960\": container with ID starting with 17b6f615a3b3bcc4841713096b843cb28b55d517000ef873d2acfb5a71060960 not found: ID does not exist" containerID="17b6f615a3b3bcc4841713096b843cb28b55d517000ef873d2acfb5a71060960" Sep 16 13:34:15.245025 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:15.244966 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b6f615a3b3bcc4841713096b843cb28b55d517000ef873d2acfb5a71060960"} err="failed to get container status \"17b6f615a3b3bcc4841713096b843cb28b55d517000ef873d2acfb5a71060960\": rpc error: code = NotFound desc = could not find container \"17b6f615a3b3bcc4841713096b843cb28b55d517000ef873d2acfb5a71060960\": container with ID starting with 17b6f615a3b3bcc4841713096b843cb28b55d517000ef873d2acfb5a71060960 not found: ID does not exist" Sep 16 13:34:15.256086 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:15.256059 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f58bd7fff-lzprj"] Sep 16 13:34:15.259907 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:15.259885 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f58bd7fff-lzprj"] Sep 16 13:34:16.353909 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:16.353864 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac39ec5a-b502-4320-9d1d-dc926718c2b2" path="/var/lib/kubelet/pods/ac39ec5a-b502-4320-9d1d-dc926718c2b2/volumes" Sep 16 13:34:18.294497 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:18.294460 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:34:18.394899 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:18.394864 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:34:25.255060 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.255020 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6fb6885f6f-7g7vp" podUID="11a17f72-796a-46e0-b646-9a194c8b4795" containerName="console" containerID="cri-o://e63aa8476cad1566ceec17d0f746f1f76cbdb07d782e25b959c4ffdbff20b47a" gracePeriod=15 Sep 16 13:34:25.485781 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.485758 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:34:25.671175 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.671083 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11a17f72-796a-46e0-b646-9a194c8b4795-console-serving-cert\") pod \"11a17f72-796a-46e0-b646-9a194c8b4795\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " Sep 16 13:34:25.671175 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.671140 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-oauth-serving-cert\") pod \"11a17f72-796a-46e0-b646-9a194c8b4795\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " Sep 16 13:34:25.671354 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.671332 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfdqn\" (UniqueName: \"kubernetes.io/projected/11a17f72-796a-46e0-b646-9a194c8b4795-kube-api-access-jfdqn\") pod \"11a17f72-796a-46e0-b646-9a194c8b4795\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " Sep 16 13:34:25.671391 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.671367 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-trusted-ca-bundle\") pod \"11a17f72-796a-46e0-b646-9a194c8b4795\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " Sep 16 13:34:25.671422 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.671396 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-console-config\") pod \"11a17f72-796a-46e0-b646-9a194c8b4795\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " Sep 16 13:34:25.671465 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.671447 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-service-ca\") pod \"11a17f72-796a-46e0-b646-9a194c8b4795\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " Sep 16 13:34:25.671519 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.671486 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11a17f72-796a-46e0-b646-9a194c8b4795-console-oauth-config\") pod \"11a17f72-796a-46e0-b646-9a194c8b4795\" (UID: \"11a17f72-796a-46e0-b646-9a194c8b4795\") " Sep 16 13:34:25.671729 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.671556 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "11a17f72-796a-46e0-b646-9a194c8b4795" (UID: "11a17f72-796a-46e0-b646-9a194c8b4795"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:34:25.671890 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.671795 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-oauth-serving-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:34:25.671890 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.671864 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-service-ca" (OuterVolumeSpecName: "service-ca") pod "11a17f72-796a-46e0-b646-9a194c8b4795" (UID: "11a17f72-796a-46e0-b646-9a194c8b4795"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:34:25.671890 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.671860 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "11a17f72-796a-46e0-b646-9a194c8b4795" (UID: "11a17f72-796a-46e0-b646-9a194c8b4795"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:34:25.672228 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.671892 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-console-config" (OuterVolumeSpecName: "console-config") pod "11a17f72-796a-46e0-b646-9a194c8b4795" (UID: "11a17f72-796a-46e0-b646-9a194c8b4795"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:34:25.673481 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.673456 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11a17f72-796a-46e0-b646-9a194c8b4795-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "11a17f72-796a-46e0-b646-9a194c8b4795" (UID: "11a17f72-796a-46e0-b646-9a194c8b4795"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:34:25.673648 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.673628 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a17f72-796a-46e0-b646-9a194c8b4795-kube-api-access-jfdqn" (OuterVolumeSpecName: "kube-api-access-jfdqn") pod "11a17f72-796a-46e0-b646-9a194c8b4795" (UID: "11a17f72-796a-46e0-b646-9a194c8b4795"). InnerVolumeSpecName "kube-api-access-jfdqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:34:25.673804 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.673778 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11a17f72-796a-46e0-b646-9a194c8b4795-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "11a17f72-796a-46e0-b646-9a194c8b4795" (UID: "11a17f72-796a-46e0-b646-9a194c8b4795"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:34:25.772771 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.772718 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfdqn\" (UniqueName: \"kubernetes.io/projected/11a17f72-796a-46e0-b646-9a194c8b4795-kube-api-access-jfdqn\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:34:25.772771 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.772766 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-trusted-ca-bundle\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:34:25.772771 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.772776 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-console-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:34:25.772771 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.772785 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11a17f72-796a-46e0-b646-9a194c8b4795-service-ca\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:34:25.773037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.772793 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11a17f72-796a-46e0-b646-9a194c8b4795-console-oauth-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:34:25.773037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:25.772801 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11a17f72-796a-46e0-b646-9a194c8b4795-console-serving-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:34:26.276461 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:26.276428 2580 generic.go:356] "Generic (PLEG): container finished" podID="11a17f72-796a-46e0-b646-9a194c8b4795" containerID="e63aa8476cad1566ceec17d0f746f1f76cbdb07d782e25b959c4ffdbff20b47a" exitCode=0 Sep 16 13:34:26.276876 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:26.276469 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fb6885f6f-7g7vp" Sep 16 13:34:26.276876 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:26.276499 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fb6885f6f-7g7vp" event={"ID":"11a17f72-796a-46e0-b646-9a194c8b4795","Type":"ContainerDied","Data":"e63aa8476cad1566ceec17d0f746f1f76cbdb07d782e25b959c4ffdbff20b47a"} Sep 16 13:34:26.276876 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:26.276539 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fb6885f6f-7g7vp" event={"ID":"11a17f72-796a-46e0-b646-9a194c8b4795","Type":"ContainerDied","Data":"e9a320e2168578a3e0357e1ec88f40d225045d6e5e2a81fcb6f2f1a604050153"} Sep 16 13:34:26.276876 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:26.276558 2580 scope.go:117] "RemoveContainer" containerID="e63aa8476cad1566ceec17d0f746f1f76cbdb07d782e25b959c4ffdbff20b47a" Sep 16 13:34:26.284947 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:26.284923 2580 scope.go:117] "RemoveContainer" containerID="e63aa8476cad1566ceec17d0f746f1f76cbdb07d782e25b959c4ffdbff20b47a" Sep 16 13:34:26.285207 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:34:26.285189 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e63aa8476cad1566ceec17d0f746f1f76cbdb07d782e25b959c4ffdbff20b47a\": container with ID starting with e63aa8476cad1566ceec17d0f746f1f76cbdb07d782e25b959c4ffdbff20b47a not found: ID does not exist" containerID="e63aa8476cad1566ceec17d0f746f1f76cbdb07d782e25b959c4ffdbff20b47a" Sep 16 13:34:26.285261 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:26.285216 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e63aa8476cad1566ceec17d0f746f1f76cbdb07d782e25b959c4ffdbff20b47a"} err="failed to get container status \"e63aa8476cad1566ceec17d0f746f1f76cbdb07d782e25b959c4ffdbff20b47a\": rpc error: code = NotFound desc = could not find container \"e63aa8476cad1566ceec17d0f746f1f76cbdb07d782e25b959c4ffdbff20b47a\": container with ID starting with e63aa8476cad1566ceec17d0f746f1f76cbdb07d782e25b959c4ffdbff20b47a not found: ID does not exist" Sep 16 13:34:26.297558 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:26.297530 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fb6885f6f-7g7vp"] Sep 16 13:34:26.303276 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:26.303251 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fb6885f6f-7g7vp"] Sep 16 13:34:26.353958 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:26.353930 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11a17f72-796a-46e0-b646-9a194c8b4795" path="/var/lib/kubelet/pods/11a17f72-796a-46e0-b646-9a194c8b4795/volumes" Sep 16 13:34:29.935444 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:29.935400 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:34:30.219036 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:34:30.218995 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:35:17.285174 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:17.285093 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:35:17.375189 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:17.375151 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:35:40.276299 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:40.276260 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:35:40.690037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:40.689954 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:35:41.958197 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:41.958157 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:35:45.378000 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:45.377963 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:35:45.549224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:45.549190 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:35:48.508627 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:48.508574 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:35:48.940137 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:48.940053 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:35:50.654172 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:50.654129 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:35:50.841963 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:50.841930 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8xcdh"] Sep 16 13:35:50.842221 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:50.842209 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac39ec5a-b502-4320-9d1d-dc926718c2b2" containerName="console" Sep 16 13:35:50.842260 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:50.842222 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac39ec5a-b502-4320-9d1d-dc926718c2b2" containerName="console" Sep 16 13:35:50.842260 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:50.842242 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11a17f72-796a-46e0-b646-9a194c8b4795" containerName="console" Sep 16 13:35:50.842260 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:50.842247 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="11a17f72-796a-46e0-b646-9a194c8b4795" containerName="console" Sep 16 13:35:50.842349 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:50.842294 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac39ec5a-b502-4320-9d1d-dc926718c2b2" containerName="console" Sep 16 13:35:50.842349 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:50.842303 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="11a17f72-796a-46e0-b646-9a194c8b4795" containerName="console" Sep 16 13:35:50.843893 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:50.843876 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-8xcdh" Sep 16 13:35:50.857744 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:50.857724 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 16 13:35:50.858990 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:50.858974 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 16 13:35:50.858990 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:50.858982 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-bzlq2\"" Sep 16 13:35:50.899322 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:50.899291 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8xcdh"] Sep 16 13:35:50.933183 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:50.933089 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lb9h\" (UniqueName: \"kubernetes.io/projected/677ce4a9-4a2a-4b33-a90f-d01155cc4b90-kube-api-access-7lb9h\") pod \"cert-manager-webhook-6486ffd997-8xcdh\" (UID: \"677ce4a9-4a2a-4b33-a90f-d01155cc4b90\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8xcdh" Sep 16 13:35:50.933183 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:50.933148 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/677ce4a9-4a2a-4b33-a90f-d01155cc4b90-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8xcdh\" (UID: \"677ce4a9-4a2a-4b33-a90f-d01155cc4b90\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8xcdh" Sep 16 13:35:51.033905 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:51.033863 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lb9h\" (UniqueName: \"kubernetes.io/projected/677ce4a9-4a2a-4b33-a90f-d01155cc4b90-kube-api-access-7lb9h\") pod \"cert-manager-webhook-6486ffd997-8xcdh\" (UID: \"677ce4a9-4a2a-4b33-a90f-d01155cc4b90\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8xcdh" Sep 16 13:35:51.034076 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:51.033956 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/677ce4a9-4a2a-4b33-a90f-d01155cc4b90-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8xcdh\" (UID: \"677ce4a9-4a2a-4b33-a90f-d01155cc4b90\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8xcdh" Sep 16 13:35:51.073434 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:51.073397 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lb9h\" (UniqueName: \"kubernetes.io/projected/677ce4a9-4a2a-4b33-a90f-d01155cc4b90-kube-api-access-7lb9h\") pod \"cert-manager-webhook-6486ffd997-8xcdh\" (UID: \"677ce4a9-4a2a-4b33-a90f-d01155cc4b90\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8xcdh" Sep 16 13:35:51.085961 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:51.085931 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/677ce4a9-4a2a-4b33-a90f-d01155cc4b90-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8xcdh\" (UID: \"677ce4a9-4a2a-4b33-a90f-d01155cc4b90\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8xcdh" Sep 16 13:35:51.152766 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:51.152731 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-8xcdh" Sep 16 13:35:51.385757 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:51.385722 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:35:51.482276 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:51.482254 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8xcdh"] Sep 16 13:35:51.484523 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:35:51.484491 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod677ce4a9_4a2a_4b33_a90f_d01155cc4b90.slice/crio-596e1ad4d8d75cde551382ac0611fbcd949c3c0a21c9797f98d001ca405320d9 WatchSource:0}: Error finding container 596e1ad4d8d75cde551382ac0611fbcd949c3c0a21c9797f98d001ca405320d9: Status 404 returned error can't find the container with id 596e1ad4d8d75cde551382ac0611fbcd949c3c0a21c9797f98d001ca405320d9 Sep 16 13:35:51.578701 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:51.578672 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-8xcdh" event={"ID":"677ce4a9-4a2a-4b33-a90f-d01155cc4b90","Type":"ContainerStarted","Data":"596e1ad4d8d75cde551382ac0611fbcd949c3c0a21c9797f98d001ca405320d9"} Sep 16 13:35:52.265818 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:52.265794 2580 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 13:35:52.397043 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:52.396895 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:35:52.624687 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:52.624584 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:35:58.944050 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:58.944002 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:35:59.613872 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:59.613776 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-8xcdh" event={"ID":"677ce4a9-4a2a-4b33-a90f-d01155cc4b90","Type":"ContainerStarted","Data":"a83424d537cf5643d11737937cd3cc3711c6a8455cb3ca13d371692c2667aa59"} Sep 16 13:35:59.613872 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:59.613841 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-8xcdh" Sep 16 13:35:59.633424 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:35:59.633381 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-8xcdh" podStartSLOduration=1.8193825110000001 podStartE2EDuration="9.6333674s" totalImagesPullingTime="7.813984889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:35:59.63300642 +0000 UTC m=+307.733819134" watchObservedRunningTime="2026-09-16 13:35:59.6333674 +0000 UTC m=+307.734180115" Sep 16 13:36:00.309246 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:00.309220 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:00.431298 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:00.431261 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:03.600044 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:03.600010 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:03.689204 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:03.689175 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:05.620207 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:05.620181 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-8xcdh" Sep 16 13:36:10.097880 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:10.097847 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:10.214746 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:10.214709 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:12.450391 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:12.450354 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:24.166454 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:24.166418 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:24.480748 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:24.480721 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:24.938997 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:24.938913 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:25.244931 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:25.244897 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:25.381745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:25.381710 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:26.658392 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:26.658346 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:26.904329 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:26.904284 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:27.058413 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.058380 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd"] Sep 16 13:36:27.062288 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.062272 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" Sep 16 13:36:27.066337 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.066316 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 16 13:36:27.066473 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.066319 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 16 13:36:27.066520 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.066489 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 16 13:36:27.067408 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.067392 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2tk7k\"" Sep 16 13:36:27.072224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.072208 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 16 13:36:27.125671 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.125637 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c703951-bae3-4dcc-a46d-f283cbc9b038-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-422kd\" (UID: \"0c703951-bae3-4dcc-a46d-f283cbc9b038\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" Sep 16 13:36:27.125822 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.125681 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0c703951-bae3-4dcc-a46d-f283cbc9b038-manifests\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-422kd\" (UID: \"0c703951-bae3-4dcc-a46d-f283cbc9b038\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" Sep 16 13:36:27.125822 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.125706 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c703951-bae3-4dcc-a46d-f283cbc9b038-tmp\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-422kd\" (UID: \"0c703951-bae3-4dcc-a46d-f283cbc9b038\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" Sep 16 13:36:27.125822 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.125741 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5p6h\" (UniqueName: \"kubernetes.io/projected/0c703951-bae3-4dcc-a46d-f283cbc9b038-kube-api-access-v5p6h\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-422kd\" (UID: \"0c703951-bae3-4dcc-a46d-f283cbc9b038\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" Sep 16 13:36:27.125822 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.125768 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0c703951-bae3-4dcc-a46d-f283cbc9b038-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-422kd\" (UID: \"0c703951-bae3-4dcc-a46d-f283cbc9b038\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" Sep 16 13:36:27.151999 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.151974 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd"] Sep 16 13:36:27.226673 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.226639 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c703951-bae3-4dcc-a46d-f283cbc9b038-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-422kd\" (UID: \"0c703951-bae3-4dcc-a46d-f283cbc9b038\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" Sep 16 13:36:27.226673 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.226676 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0c703951-bae3-4dcc-a46d-f283cbc9b038-manifests\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-422kd\" (UID: \"0c703951-bae3-4dcc-a46d-f283cbc9b038\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" Sep 16 13:36:27.226883 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.226697 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c703951-bae3-4dcc-a46d-f283cbc9b038-tmp\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-422kd\" (UID: \"0c703951-bae3-4dcc-a46d-f283cbc9b038\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" Sep 16 13:36:27.226883 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.226725 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5p6h\" (UniqueName: \"kubernetes.io/projected/0c703951-bae3-4dcc-a46d-f283cbc9b038-kube-api-access-v5p6h\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-422kd\" (UID: \"0c703951-bae3-4dcc-a46d-f283cbc9b038\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" Sep 16 13:36:27.226883 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.226752 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0c703951-bae3-4dcc-a46d-f283cbc9b038-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-422kd\" (UID: \"0c703951-bae3-4dcc-a46d-f283cbc9b038\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" Sep 16 13:36:27.227210 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.227176 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0c703951-bae3-4dcc-a46d-f283cbc9b038-manifests\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-422kd\" (UID: \"0c703951-bae3-4dcc-a46d-f283cbc9b038\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" Sep 16 13:36:27.229264 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.229237 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c703951-bae3-4dcc-a46d-f283cbc9b038-tmp\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-422kd\" (UID: \"0c703951-bae3-4dcc-a46d-f283cbc9b038\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" Sep 16 13:36:27.229406 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.229390 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c703951-bae3-4dcc-a46d-f283cbc9b038-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-422kd\" (UID: \"0c703951-bae3-4dcc-a46d-f283cbc9b038\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" Sep 16 13:36:27.229528 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.229509 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0c703951-bae3-4dcc-a46d-f283cbc9b038-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-422kd\" (UID: \"0c703951-bae3-4dcc-a46d-f283cbc9b038\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" Sep 16 13:36:27.239867 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.239843 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5p6h\" (UniqueName: \"kubernetes.io/projected/0c703951-bae3-4dcc-a46d-f283cbc9b038-kube-api-access-v5p6h\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-422kd\" (UID: \"0c703951-bae3-4dcc-a46d-f283cbc9b038\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" Sep 16 13:36:27.371852 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.371754 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" Sep 16 13:36:27.380732 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.380700 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:27.660432 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.660331 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:27.677105 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.677076 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd"] Sep 16 13:36:27.680008 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:36:27.679982 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c703951_bae3_4dcc_a46d_f283cbc9b038.slice/crio-42191f7d9dec3f375d86233d33c61c405116a4ae1297845c10b26c5d26bf137f WatchSource:0}: Error finding container 42191f7d9dec3f375d86233d33c61c405116a4ae1297845c10b26c5d26bf137f: Status 404 returned error can't find the container with id 42191f7d9dec3f375d86233d33c61c405116a4ae1297845c10b26c5d26bf137f Sep 16 13:36:27.681715 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.681697 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 13:36:27.717147 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.717113 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" event={"ID":"0c703951-bae3-4dcc-a46d-f283cbc9b038","Type":"ContainerStarted","Data":"42191f7d9dec3f375d86233d33c61c405116a4ae1297845c10b26c5d26bf137f"} Sep 16 13:36:27.845023 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:27.844992 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:28.006241 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:28.006205 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:28.168196 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:28.168163 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:30.730927 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:30.730894 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" event={"ID":"0c703951-bae3-4dcc-a46d-f283cbc9b038","Type":"ContainerStarted","Data":"05c430e4ff8de78d979901446ac1ffd81f83ee1cf2ff309f91b7878bbe3654a1"} Sep 16 13:36:31.735705 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:31.735672 2580 generic.go:356] "Generic (PLEG): container finished" podID="0c703951-bae3-4dcc-a46d-f283cbc9b038" containerID="05c430e4ff8de78d979901446ac1ffd81f83ee1cf2ff309f91b7878bbe3654a1" exitCode=0 Sep 16 13:36:31.736167 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:31.735758 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" event={"ID":"0c703951-bae3-4dcc-a46d-f283cbc9b038","Type":"ContainerDied","Data":"05c430e4ff8de78d979901446ac1ffd81f83ee1cf2ff309f91b7878bbe3654a1"} Sep 16 13:36:32.741017 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:32.740981 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" event={"ID":"0c703951-bae3-4dcc-a46d-f283cbc9b038","Type":"ContainerStarted","Data":"eed561df499437aef9decc72021d3e427c362a931ec7b8abb4b1269140d15b32"} Sep 16 13:36:32.741400 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:32.741218 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" Sep 16 13:36:32.807196 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:32.807127 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" podStartSLOduration=2.4027687220000002 podStartE2EDuration="6.807105209s" totalImagesPullingTime="3.353488152s" totalInitContainerRuntime="1.050848335s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:36:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:36:32.804317682 +0000 UTC m=+340.905130396" watchObservedRunningTime="2026-09-16 13:36:32.807105209 +0000 UTC m=+340.907917924" Sep 16 13:36:37.155435 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:37.155398 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:37.290473 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:37.290439 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:38.885824 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:38.885790 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:39.071060 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:39.071022 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:39.298111 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:39.298077 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:39.444119 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:39.444084 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:43.747922 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:43.747850 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-422kd" Sep 16 13:36:45.072331 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.072300 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8dd669f78-5qbld"] Sep 16 13:36:45.075475 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.075458 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8dd669f78-5qbld" Sep 16 13:36:45.079481 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.079397 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 16 13:36:45.079481 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.079463 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-x64lh\"" Sep 16 13:36:45.079755 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.079473 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 16 13:36:45.079755 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.079712 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:36:45.079755 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.079725 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 16 13:36:45.079755 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.079726 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 16 13:36:45.092036 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.092016 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8dd669f78-5qbld"] Sep 16 13:36:45.180335 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.180300 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/66a17b7a-d11b-425c-a11f-3b581af44124-metrics-cert\") pod \"lws-controller-manager-8dd669f78-5qbld\" (UID: \"66a17b7a-d11b-425c-a11f-3b581af44124\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-5qbld" Sep 16 13:36:45.180498 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.180353 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66a17b7a-d11b-425c-a11f-3b581af44124-cert\") pod \"lws-controller-manager-8dd669f78-5qbld\" (UID: \"66a17b7a-d11b-425c-a11f-3b581af44124\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-5qbld" Sep 16 13:36:45.180498 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.180400 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htr9f\" (UniqueName: \"kubernetes.io/projected/66a17b7a-d11b-425c-a11f-3b581af44124-kube-api-access-htr9f\") pod \"lws-controller-manager-8dd669f78-5qbld\" (UID: \"66a17b7a-d11b-425c-a11f-3b581af44124\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-5qbld" Sep 16 13:36:45.180498 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.180425 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/66a17b7a-d11b-425c-a11f-3b581af44124-manager-config\") pod \"lws-controller-manager-8dd669f78-5qbld\" (UID: \"66a17b7a-d11b-425c-a11f-3b581af44124\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-5qbld" Sep 16 13:36:45.281068 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.281020 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66a17b7a-d11b-425c-a11f-3b581af44124-cert\") pod \"lws-controller-manager-8dd669f78-5qbld\" (UID: \"66a17b7a-d11b-425c-a11f-3b581af44124\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-5qbld" Sep 16 13:36:45.281068 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.281077 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htr9f\" (UniqueName: \"kubernetes.io/projected/66a17b7a-d11b-425c-a11f-3b581af44124-kube-api-access-htr9f\") pod \"lws-controller-manager-8dd669f78-5qbld\" (UID: \"66a17b7a-d11b-425c-a11f-3b581af44124\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-5qbld" Sep 16 13:36:45.281318 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.281103 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/66a17b7a-d11b-425c-a11f-3b581af44124-manager-config\") pod \"lws-controller-manager-8dd669f78-5qbld\" (UID: \"66a17b7a-d11b-425c-a11f-3b581af44124\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-5qbld" Sep 16 13:36:45.281318 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.281136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/66a17b7a-d11b-425c-a11f-3b581af44124-metrics-cert\") pod \"lws-controller-manager-8dd669f78-5qbld\" (UID: \"66a17b7a-d11b-425c-a11f-3b581af44124\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-5qbld" Sep 16 13:36:45.281890 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.281871 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/66a17b7a-d11b-425c-a11f-3b581af44124-manager-config\") pod \"lws-controller-manager-8dd669f78-5qbld\" (UID: \"66a17b7a-d11b-425c-a11f-3b581af44124\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-5qbld" Sep 16 13:36:45.283711 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.283688 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66a17b7a-d11b-425c-a11f-3b581af44124-cert\") pod \"lws-controller-manager-8dd669f78-5qbld\" (UID: \"66a17b7a-d11b-425c-a11f-3b581af44124\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-5qbld" Sep 16 13:36:45.283909 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.283886 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/66a17b7a-d11b-425c-a11f-3b581af44124-metrics-cert\") pod \"lws-controller-manager-8dd669f78-5qbld\" (UID: \"66a17b7a-d11b-425c-a11f-3b581af44124\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-5qbld" Sep 16 13:36:45.289543 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.289521 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htr9f\" (UniqueName: \"kubernetes.io/projected/66a17b7a-d11b-425c-a11f-3b581af44124-kube-api-access-htr9f\") pod \"lws-controller-manager-8dd669f78-5qbld\" (UID: \"66a17b7a-d11b-425c-a11f-3b581af44124\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-5qbld" Sep 16 13:36:45.385004 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.384926 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8dd669f78-5qbld" Sep 16 13:36:45.674693 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.674616 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8dd669f78-5qbld"] Sep 16 13:36:45.680215 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:36:45.680188 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a17b7a_d11b_425c_a11f_3b581af44124.slice/crio-072137a467d6ce22d96a2d3ce05703eb7055c9c0408b15e529ff3d906ffb5557 WatchSource:0}: Error finding container 072137a467d6ce22d96a2d3ce05703eb7055c9c0408b15e529ff3d906ffb5557: Status 404 returned error can't find the container with id 072137a467d6ce22d96a2d3ce05703eb7055c9c0408b15e529ff3d906ffb5557 Sep 16 13:36:45.796993 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:45.796962 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8dd669f78-5qbld" event={"ID":"66a17b7a-d11b-425c-a11f-3b581af44124","Type":"ContainerStarted","Data":"072137a467d6ce22d96a2d3ce05703eb7055c9c0408b15e529ff3d906ffb5557"} Sep 16 13:36:46.092163 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:46.092113 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:46.228223 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:46.228186 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:47.910945 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:47.910902 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:47.973467 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:47.973427 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7b66dcfb4c-h7zvs"] Sep 16 13:36:47.976918 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:47.976895 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b66dcfb4c-h7zvs" Sep 16 13:36:47.979661 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:47.979634 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 16 13:36:47.980082 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:47.980064 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-kx8kk\"" Sep 16 13:36:47.980868 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:47.980848 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 16 13:36:47.980972 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:47.980922 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 16 13:36:47.980972 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:47.980927 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 16 13:36:47.990970 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:47.990946 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b66dcfb4c-h7zvs"] Sep 16 13:36:48.104063 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:48.104019 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4807f9b2-4daa-475e-95f8-3c25029887d5-tls-certs\") pod \"kube-auth-proxy-7b66dcfb4c-h7zvs\" (UID: \"4807f9b2-4daa-475e-95f8-3c25029887d5\") " pod="openshift-ingress/kube-auth-proxy-7b66dcfb4c-h7zvs" Sep 16 13:36:48.104245 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:48.104100 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2x9f\" (UniqueName: \"kubernetes.io/projected/4807f9b2-4daa-475e-95f8-3c25029887d5-kube-api-access-n2x9f\") pod \"kube-auth-proxy-7b66dcfb4c-h7zvs\" (UID: \"4807f9b2-4daa-475e-95f8-3c25029887d5\") " pod="openshift-ingress/kube-auth-proxy-7b66dcfb4c-h7zvs" Sep 16 13:36:48.104245 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:48.104120 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4807f9b2-4daa-475e-95f8-3c25029887d5-tmp\") pod \"kube-auth-proxy-7b66dcfb4c-h7zvs\" (UID: \"4807f9b2-4daa-475e-95f8-3c25029887d5\") " pod="openshift-ingress/kube-auth-proxy-7b66dcfb4c-h7zvs" Sep 16 13:36:48.205350 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:48.205271 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4807f9b2-4daa-475e-95f8-3c25029887d5-tls-certs\") pod \"kube-auth-proxy-7b66dcfb4c-h7zvs\" (UID: \"4807f9b2-4daa-475e-95f8-3c25029887d5\") " pod="openshift-ingress/kube-auth-proxy-7b66dcfb4c-h7zvs" Sep 16 13:36:48.205350 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:48.205325 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2x9f\" (UniqueName: \"kubernetes.io/projected/4807f9b2-4daa-475e-95f8-3c25029887d5-kube-api-access-n2x9f\") pod \"kube-auth-proxy-7b66dcfb4c-h7zvs\" (UID: \"4807f9b2-4daa-475e-95f8-3c25029887d5\") " pod="openshift-ingress/kube-auth-proxy-7b66dcfb4c-h7zvs" Sep 16 13:36:48.205350 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:48.205343 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4807f9b2-4daa-475e-95f8-3c25029887d5-tmp\") pod \"kube-auth-proxy-7b66dcfb4c-h7zvs\" (UID: \"4807f9b2-4daa-475e-95f8-3c25029887d5\") " pod="openshift-ingress/kube-auth-proxy-7b66dcfb4c-h7zvs" Sep 16 13:36:48.207734 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:48.207699 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4807f9b2-4daa-475e-95f8-3c25029887d5-tmp\") pod \"kube-auth-proxy-7b66dcfb4c-h7zvs\" (UID: \"4807f9b2-4daa-475e-95f8-3c25029887d5\") " pod="openshift-ingress/kube-auth-proxy-7b66dcfb4c-h7zvs" Sep 16 13:36:48.208060 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:48.208038 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4807f9b2-4daa-475e-95f8-3c25029887d5-tls-certs\") pod \"kube-auth-proxy-7b66dcfb4c-h7zvs\" (UID: \"4807f9b2-4daa-475e-95f8-3c25029887d5\") " pod="openshift-ingress/kube-auth-proxy-7b66dcfb4c-h7zvs" Sep 16 13:36:48.215146 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:48.215121 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2x9f\" (UniqueName: \"kubernetes.io/projected/4807f9b2-4daa-475e-95f8-3c25029887d5-kube-api-access-n2x9f\") pod \"kube-auth-proxy-7b66dcfb4c-h7zvs\" (UID: \"4807f9b2-4daa-475e-95f8-3c25029887d5\") " pod="openshift-ingress/kube-auth-proxy-7b66dcfb4c-h7zvs" Sep 16 13:36:48.264622 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:48.264583 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:48.289331 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:48.289303 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b66dcfb4c-h7zvs" Sep 16 13:36:48.582033 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:48.581979 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b66dcfb4c-h7zvs"] Sep 16 13:36:48.584683 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:36:48.584656 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4807f9b2_4daa_475e_95f8_3c25029887d5.slice/crio-fcc8c5d2a2fadf3b74e4ea8f2faede81c549fb2494425e256b8fc18c060c0129 WatchSource:0}: Error finding container fcc8c5d2a2fadf3b74e4ea8f2faede81c549fb2494425e256b8fc18c060c0129: Status 404 returned error can't find the container with id fcc8c5d2a2fadf3b74e4ea8f2faede81c549fb2494425e256b8fc18c060c0129 Sep 16 13:36:48.812063 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:48.812024 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8dd669f78-5qbld" event={"ID":"66a17b7a-d11b-425c-a11f-3b581af44124","Type":"ContainerStarted","Data":"589e9f1e4448d3eb1fd48713b82a247f7c97056a41b4ab97eb5b41a554d52183"} Sep 16 13:36:48.812226 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:48.812131 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8dd669f78-5qbld" Sep 16 13:36:48.813196 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:48.813170 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b66dcfb4c-h7zvs" event={"ID":"4807f9b2-4daa-475e-95f8-3c25029887d5","Type":"ContainerStarted","Data":"fcc8c5d2a2fadf3b74e4ea8f2faede81c549fb2494425e256b8fc18c060c0129"} Sep 16 13:36:48.846159 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:48.846074 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8dd669f78-5qbld" podStartSLOduration=1.283334214 podStartE2EDuration="3.846057052s" totalImagesPullingTime="2.562722838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:36:48.84452283 +0000 UTC m=+356.945335546" watchObservedRunningTime="2026-09-16 13:36:48.846057052 +0000 UTC m=+356.946869767" Sep 16 13:36:49.787759 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:49.787716 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:49.885774 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:49.885736 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:51.661779 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:51.661751 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:52.831582 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:52.831544 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b66dcfb4c-h7zvs" event={"ID":"4807f9b2-4daa-475e-95f8-3c25029887d5","Type":"ContainerStarted","Data":"66132183f4aafc185c52246022967848c8fbf59a63bbfab0a13231ea0a17b661"} Sep 16 13:36:52.854528 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:52.854474 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7b66dcfb4c-h7zvs" podStartSLOduration=2.506965102 podStartE2EDuration="5.854458333s" totalImagesPullingTime="3.347493231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:36:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:36:52.853580019 +0000 UTC m=+360.954392733" watchObservedRunningTime="2026-09-16 13:36:52.854458333 +0000 UTC m=+360.955271047" Sep 16 13:36:55.505028 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:55.504995 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:55.618496 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:55.618463 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:55.894279 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:55.894201 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:56.108265 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:56.108232 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:57.394388 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:57.394354 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:57.550183 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:57.550149 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:57.729662 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:57.729630 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:58.036247 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:58.036173 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:58.328704 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:58.328626 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:59.820530 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:36:59.820482 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8dd669f78-5qbld" Sep 16 13:37:00.063183 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:00.063145 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:00.168737 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:00.168637 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:02.000125 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:02.000091 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:03.310155 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:03.310114 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:03.420866 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:03.420836 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:08.709325 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:08.709280 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:08.931253 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:08.931225 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:09.111989 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:09.111954 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:10.368219 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:10.368184 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:10.778639 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:10.778591 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:10.781334 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:10.781308 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dc777cfcf-vxwsq"] Sep 16 13:37:11.075929 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:11.075849 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:11.228106 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:11.228072 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:35.800924 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:35.800859 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7dc777cfcf-vxwsq" podUID="19d8caf0-7b53-459c-a11b-d4a56bc28b7b" containerName="console" containerID="cri-o://09cd445fca4e056aab3e1840767ca053316efebb9ad5c168b2eb994b7fdca8bc" gracePeriod=15 Sep 16 13:37:36.008746 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.008716 2580 generic.go:356] "Generic (PLEG): container finished" podID="19d8caf0-7b53-459c-a11b-d4a56bc28b7b" containerID="09cd445fca4e056aab3e1840767ca053316efebb9ad5c168b2eb994b7fdca8bc" exitCode=0 Sep 16 13:37:36.008902 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.008783 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc777cfcf-vxwsq" event={"ID":"19d8caf0-7b53-459c-a11b-d4a56bc28b7b","Type":"ContainerDied","Data":"09cd445fca4e056aab3e1840767ca053316efebb9ad5c168b2eb994b7fdca8bc"} Sep 16 13:37:36.035274 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.035250 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:37:36.107255 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.107173 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-trusted-ca-bundle\") pod \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " Sep 16 13:37:36.107255 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.107209 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c92j8\" (UniqueName: \"kubernetes.io/projected/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-kube-api-access-c92j8\") pod \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " Sep 16 13:37:36.107255 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.107240 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-console-oauth-config\") pod \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " Sep 16 13:37:36.107517 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.107277 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-console-config\") pod \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " Sep 16 13:37:36.107517 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.107319 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-service-ca\") pod \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " Sep 16 13:37:36.107517 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.107353 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-console-serving-cert\") pod \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " Sep 16 13:37:36.107517 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.107381 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-oauth-serving-cert\") pod \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\" (UID: \"19d8caf0-7b53-459c-a11b-d4a56bc28b7b\") " Sep 16 13:37:36.107933 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.107895 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "19d8caf0-7b53-459c-a11b-d4a56bc28b7b" (UID: "19d8caf0-7b53-459c-a11b-d4a56bc28b7b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:37:36.107933 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.107823 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-console-config" (OuterVolumeSpecName: "console-config") pod "19d8caf0-7b53-459c-a11b-d4a56bc28b7b" (UID: "19d8caf0-7b53-459c-a11b-d4a56bc28b7b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:37:36.107933 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.107833 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-service-ca" (OuterVolumeSpecName: "service-ca") pod "19d8caf0-7b53-459c-a11b-d4a56bc28b7b" (UID: "19d8caf0-7b53-459c-a11b-d4a56bc28b7b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:37:36.108349 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.108004 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "19d8caf0-7b53-459c-a11b-d4a56bc28b7b" (UID: "19d8caf0-7b53-459c-a11b-d4a56bc28b7b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:37:36.109650 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.109622 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "19d8caf0-7b53-459c-a11b-d4a56bc28b7b" (UID: "19d8caf0-7b53-459c-a11b-d4a56bc28b7b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:37:36.109761 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.109676 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-kube-api-access-c92j8" (OuterVolumeSpecName: "kube-api-access-c92j8") pod "19d8caf0-7b53-459c-a11b-d4a56bc28b7b" (UID: "19d8caf0-7b53-459c-a11b-d4a56bc28b7b"). InnerVolumeSpecName "kube-api-access-c92j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:37:36.109804 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.109755 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "19d8caf0-7b53-459c-a11b-d4a56bc28b7b" (UID: "19d8caf0-7b53-459c-a11b-d4a56bc28b7b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:37:36.208451 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.208404 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-service-ca\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:37:36.208451 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.208446 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-console-serving-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:37:36.208451 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.208456 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-oauth-serving-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:37:36.208451 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.208464 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-trusted-ca-bundle\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:37:36.208755 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.208479 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c92j8\" (UniqueName: \"kubernetes.io/projected/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-kube-api-access-c92j8\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:37:36.208755 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.208488 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-console-oauth-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:37:36.208755 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:36.208496 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19d8caf0-7b53-459c-a11b-d4a56bc28b7b-console-config\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:37:37.014461 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:37.014435 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc777cfcf-vxwsq" Sep 16 13:37:37.014461 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:37.014435 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc777cfcf-vxwsq" event={"ID":"19d8caf0-7b53-459c-a11b-d4a56bc28b7b","Type":"ContainerDied","Data":"2e01d668894dee2e282b6d53de5b13e66fca2f0d3f8e8ffe032dd486c6200836"} Sep 16 13:37:37.014995 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:37.014504 2580 scope.go:117] "RemoveContainer" containerID="09cd445fca4e056aab3e1840767ca053316efebb9ad5c168b2eb994b7fdca8bc" Sep 16 13:37:37.035101 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:37.035075 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dc777cfcf-vxwsq"] Sep 16 13:37:37.037270 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:37.037249 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7dc777cfcf-vxwsq"] Sep 16 13:37:38.354780 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:38.354751 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19d8caf0-7b53-459c-a11b-d4a56bc28b7b" path="/var/lib/kubelet/pods/19d8caf0-7b53-459c-a11b-d4a56bc28b7b/volumes" Sep 16 13:37:58.457838 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:58.457803 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:58.552374 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:37:58.552340 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:13.911016 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:13.910942 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:14.684423 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:14.684389 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:15.481555 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:15.481510 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:18.645108 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:18.645070 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:18.776419 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:18.776378 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:19.110058 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:19.110024 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:20.187063 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:20.187026 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:20.320983 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:20.320955 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:21.731370 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:21.731341 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:21.825419 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:21.825391 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:23.942709 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:23.942637 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:24.570009 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:24.569978 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:25.382763 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:25.382728 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:26.089780 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.089748 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:26.123532 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.123499 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-th4jv"] Sep 16 13:38:26.123829 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.123816 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19d8caf0-7b53-459c-a11b-d4a56bc28b7b" containerName="console" Sep 16 13:38:26.123877 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.123832 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="19d8caf0-7b53-459c-a11b-d4a56bc28b7b" containerName="console" Sep 16 13:38:26.123910 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.123885 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="19d8caf0-7b53-459c-a11b-d4a56bc28b7b" containerName="console" Sep 16 13:38:26.125500 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.125482 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-th4jv" Sep 16 13:38:26.129173 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.129154 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 16 13:38:26.129445 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.129428 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 16 13:38:26.129542 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.129527 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 16 13:38:26.130424 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.130405 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 16 13:38:26.130424 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.130413 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8r4wt\"" Sep 16 13:38:26.139952 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.139930 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-th4jv"] Sep 16 13:38:26.306910 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.306872 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnq2x\" (UniqueName: \"kubernetes.io/projected/0f95b231-d809-49a0-bdbd-5b711ab3a87f-kube-api-access-pnq2x\") pod \"kuadrant-console-plugin-fbdd4f445-th4jv\" (UID: \"0f95b231-d809-49a0-bdbd-5b711ab3a87f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-th4jv" Sep 16 13:38:26.307074 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.306948 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f95b231-d809-49a0-bdbd-5b711ab3a87f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-th4jv\" (UID: \"0f95b231-d809-49a0-bdbd-5b711ab3a87f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-th4jv" Sep 16 13:38:26.307074 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.306992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f95b231-d809-49a0-bdbd-5b711ab3a87f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-th4jv\" (UID: \"0f95b231-d809-49a0-bdbd-5b711ab3a87f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-th4jv" Sep 16 13:38:26.408308 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.408218 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f95b231-d809-49a0-bdbd-5b711ab3a87f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-th4jv\" (UID: \"0f95b231-d809-49a0-bdbd-5b711ab3a87f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-th4jv" Sep 16 13:38:26.408308 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.408260 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f95b231-d809-49a0-bdbd-5b711ab3a87f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-th4jv\" (UID: \"0f95b231-d809-49a0-bdbd-5b711ab3a87f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-th4jv" Sep 16 13:38:26.408308 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.408301 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnq2x\" (UniqueName: \"kubernetes.io/projected/0f95b231-d809-49a0-bdbd-5b711ab3a87f-kube-api-access-pnq2x\") pod \"kuadrant-console-plugin-fbdd4f445-th4jv\" (UID: \"0f95b231-d809-49a0-bdbd-5b711ab3a87f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-th4jv" Sep 16 13:38:26.408774 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:38:26.408376 2580 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Sep 16 13:38:26.408774 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:38:26.408471 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f95b231-d809-49a0-bdbd-5b711ab3a87f-plugin-serving-cert podName:0f95b231-d809-49a0-bdbd-5b711ab3a87f nodeName:}" failed. No retries permitted until 2026-09-16 13:38:26.90845093 +0000 UTC m=+455.009263625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/0f95b231-d809-49a0-bdbd-5b711ab3a87f-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-th4jv" (UID: "0f95b231-d809-49a0-bdbd-5b711ab3a87f") : secret "plugin-serving-cert" not found Sep 16 13:38:26.408869 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.408849 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f95b231-d809-49a0-bdbd-5b711ab3a87f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-th4jv\" (UID: \"0f95b231-d809-49a0-bdbd-5b711ab3a87f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-th4jv" Sep 16 13:38:26.420161 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.420127 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnq2x\" (UniqueName: \"kubernetes.io/projected/0f95b231-d809-49a0-bdbd-5b711ab3a87f-kube-api-access-pnq2x\") pod \"kuadrant-console-plugin-fbdd4f445-th4jv\" (UID: \"0f95b231-d809-49a0-bdbd-5b711ab3a87f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-th4jv" Sep 16 13:38:26.763568 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.763529 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:26.912935 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.912893 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f95b231-d809-49a0-bdbd-5b711ab3a87f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-th4jv\" (UID: \"0f95b231-d809-49a0-bdbd-5b711ab3a87f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-th4jv" Sep 16 13:38:26.915396 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:26.915373 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f95b231-d809-49a0-bdbd-5b711ab3a87f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-th4jv\" (UID: \"0f95b231-d809-49a0-bdbd-5b711ab3a87f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-th4jv" Sep 16 13:38:27.034133 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:27.034038 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-th4jv" Sep 16 13:38:27.318597 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:27.318518 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-th4jv"] Sep 16 13:38:27.321980 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:38:27.321956 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f95b231_d809_49a0_bdbd_5b711ab3a87f.slice/crio-c5aa61697dc67b241891c068e33526769174d2b7f0a3e872b78d7b5d07543b23 WatchSource:0}: Error finding container c5aa61697dc67b241891c068e33526769174d2b7f0a3e872b78d7b5d07543b23: Status 404 returned error can't find the container with id c5aa61697dc67b241891c068e33526769174d2b7f0a3e872b78d7b5d07543b23 Sep 16 13:38:28.208503 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:28.208414 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-th4jv" event={"ID":"0f95b231-d809-49a0-bdbd-5b711ab3a87f","Type":"ContainerStarted","Data":"c5aa61697dc67b241891c068e33526769174d2b7f0a3e872b78d7b5d07543b23"} Sep 16 13:38:29.159041 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:29.159005 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:29.315586 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:29.315548 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:29.576600 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:29.576549 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:29.676092 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:29.676054 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:29.928640 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:29.928468 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:30.012694 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:30.012641 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:32.231962 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:32.231925 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-th4jv" event={"ID":"0f95b231-d809-49a0-bdbd-5b711ab3a87f","Type":"ContainerStarted","Data":"bfe064607d3e07602b86ec027b5be09facdf5d4664a04a3564739a2f5de06d90"} Sep 16 13:38:38.289619 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:38.289568 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:38.429266 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:38.429235 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:38.765881 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:38.765832 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:40.684731 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:40.684693 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:40.906210 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:40.906172 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:45.311781 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:45.311747 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:45.412891 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:45.412859 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:46.500645 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:46.500590 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:46.606812 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:46.606780 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:50.526417 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:50.526373 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:50.633937 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:50.633905 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:50.744125 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:50.744073 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:50.845925 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:50.845844 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:50.949450 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:50.949421 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:51.048028 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:51.047986 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:55.030172 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:55.030129 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:55.220904 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:55.220870 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:59.101451 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:59.101413 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:59.206236 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:38:59.206202 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:01.604756 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:01.604704 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:01.704227 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:01.704201 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:06.303577 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:06.303543 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:06.424851 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:06.424820 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:09.487463 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:09.487420 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:09.687593 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:09.687554 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:11.969535 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:11.969505 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:12.066296 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.066266 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:12.130167 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.130136 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:12.311955 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.311856 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-th4jv" podStartSLOduration=42.062710457 podStartE2EDuration="46.311839146s" totalImagesPullingTime="4.249128689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:38:32.263638415 +0000 UTC m=+460.364451132" watchObservedRunningTime="2026-09-16 13:39:12.311839146 +0000 UTC m=+500.412651860" Sep 16 13:39:12.312351 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.312336 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:39:12.331561 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.331533 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:39:12.331728 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.331646 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-f8lqw" Sep 16 13:39:12.334550 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.334528 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 16 13:39:12.340407 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.340383 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:39:12.477057 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.477016 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c412e0bc-13ed-48c9-b3be-c8350310b2ae-config-file\") pod \"limitador-limitador-84994f7ffc-f8lqw\" (UID: \"c412e0bc-13ed-48c9-b3be-c8350310b2ae\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-f8lqw" Sep 16 13:39:12.477057 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.477063 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8sfc\" (UniqueName: \"kubernetes.io/projected/c412e0bc-13ed-48c9-b3be-c8350310b2ae-kube-api-access-f8sfc\") pod \"limitador-limitador-84994f7ffc-f8lqw\" (UID: \"c412e0bc-13ed-48c9-b3be-c8350310b2ae\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-f8lqw" Sep 16 13:39:12.577981 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.577893 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c412e0bc-13ed-48c9-b3be-c8350310b2ae-config-file\") pod \"limitador-limitador-84994f7ffc-f8lqw\" (UID: \"c412e0bc-13ed-48c9-b3be-c8350310b2ae\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-f8lqw" Sep 16 13:39:12.577981 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.577943 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8sfc\" (UniqueName: \"kubernetes.io/projected/c412e0bc-13ed-48c9-b3be-c8350310b2ae-kube-api-access-f8sfc\") pod \"limitador-limitador-84994f7ffc-f8lqw\" (UID: \"c412e0bc-13ed-48c9-b3be-c8350310b2ae\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-f8lqw" Sep 16 13:39:12.578533 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.578510 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c412e0bc-13ed-48c9-b3be-c8350310b2ae-config-file\") pod \"limitador-limitador-84994f7ffc-f8lqw\" (UID: \"c412e0bc-13ed-48c9-b3be-c8350310b2ae\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-f8lqw" Sep 16 13:39:12.587358 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.587329 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8sfc\" (UniqueName: \"kubernetes.io/projected/c412e0bc-13ed-48c9-b3be-c8350310b2ae-kube-api-access-f8sfc\") pod \"limitador-limitador-84994f7ffc-f8lqw\" (UID: \"c412e0bc-13ed-48c9-b3be-c8350310b2ae\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-f8lqw" Sep 16 13:39:12.642533 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.642503 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-f8lqw" Sep 16 13:39:12.786007 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.785973 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zqmlz"] Sep 16 13:39:12.789507 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.789485 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-zqmlz" Sep 16 13:39:12.795724 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.795700 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xfzjj\"" Sep 16 13:39:12.807592 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.807564 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zqmlz"] Sep 16 13:39:12.876274 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.876192 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:12.881354 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.881336 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnbwt\" (UniqueName: \"kubernetes.io/projected/f8b40a8f-e3d9-4d60-bb3e-825fcbf324df-kube-api-access-vnbwt\") pod \"authorino-7bcfd8f9cd-zqmlz\" (UID: \"f8b40a8f-e3d9-4d60-bb3e-825fcbf324df\") " pod="kuadrant-system/authorino-7bcfd8f9cd-zqmlz" Sep 16 13:39:12.914085 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.914049 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:39:12.917072 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:39:12.917043 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc412e0bc_13ed_48c9_b3be_c8350310b2ae.slice/crio-9acc9ce4f3a99a29d63930b5f3570416d50aa90e5d57c5684772f9aebcc62567 WatchSource:0}: Error finding container 9acc9ce4f3a99a29d63930b5f3570416d50aa90e5d57c5684772f9aebcc62567: Status 404 returned error can't find the container with id 9acc9ce4f3a99a29d63930b5f3570416d50aa90e5d57c5684772f9aebcc62567 Sep 16 13:39:12.963266 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.963232 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:12.982334 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.982302 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnbwt\" (UniqueName: \"kubernetes.io/projected/f8b40a8f-e3d9-4d60-bb3e-825fcbf324df-kube-api-access-vnbwt\") pod \"authorino-7bcfd8f9cd-zqmlz\" (UID: \"f8b40a8f-e3d9-4d60-bb3e-825fcbf324df\") " pod="kuadrant-system/authorino-7bcfd8f9cd-zqmlz" Sep 16 13:39:12.991134 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:12.991113 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnbwt\" (UniqueName: \"kubernetes.io/projected/f8b40a8f-e3d9-4d60-bb3e-825fcbf324df-kube-api-access-vnbwt\") pod \"authorino-7bcfd8f9cd-zqmlz\" (UID: \"f8b40a8f-e3d9-4d60-bb3e-825fcbf324df\") " pod="kuadrant-system/authorino-7bcfd8f9cd-zqmlz" Sep 16 13:39:13.099234 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:13.099191 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-zqmlz" Sep 16 13:39:13.369199 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:13.369175 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zqmlz"] Sep 16 13:39:13.371578 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:39:13.371559 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8b40a8f_e3d9_4d60_bb3e_825fcbf324df.slice/crio-f0066a12a03fdf42dd6676fa627b33c851bfa06af1f49e540f7d97df58039996 WatchSource:0}: Error finding container f0066a12a03fdf42dd6676fa627b33c851bfa06af1f49e540f7d97df58039996: Status 404 returned error can't find the container with id f0066a12a03fdf42dd6676fa627b33c851bfa06af1f49e540f7d97df58039996 Sep 16 13:39:13.387284 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:13.387252 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-f8lqw" event={"ID":"c412e0bc-13ed-48c9-b3be-c8350310b2ae","Type":"ContainerStarted","Data":"9acc9ce4f3a99a29d63930b5f3570416d50aa90e5d57c5684772f9aebcc62567"} Sep 16 13:39:13.388408 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:13.388380 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-zqmlz" event={"ID":"f8b40a8f-e3d9-4d60-bb3e-825fcbf324df","Type":"ContainerStarted","Data":"f0066a12a03fdf42dd6676fa627b33c851bfa06af1f49e540f7d97df58039996"} Sep 16 13:39:15.997385 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:15.997341 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zqmlz"] Sep 16 13:39:16.404462 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:16.404420 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-f8lqw" event={"ID":"c412e0bc-13ed-48c9-b3be-c8350310b2ae","Type":"ContainerStarted","Data":"a84c5543e8fc0e776e0bcfd366dbdde6be83f96a8cb8d742f22390c4b225c7c1"} Sep 16 13:39:16.404775 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:16.404585 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-f8lqw" Sep 16 13:39:16.406006 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:16.405983 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-zqmlz" event={"ID":"f8b40a8f-e3d9-4d60-bb3e-825fcbf324df","Type":"ContainerStarted","Data":"00412ac98f314589d951b8b7918a18bcd59c79ad2c48b90d3a75e57b882c7dd9"} Sep 16 13:39:16.424757 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:16.424703 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-f8lqw" podStartSLOduration=1.56679507 podStartE2EDuration="4.424686101s" totalImagesPullingTime="2.857891031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:39:16.424378329 +0000 UTC m=+504.525191045" watchObservedRunningTime="2026-09-16 13:39:16.424686101 +0000 UTC m=+504.525498815" Sep 16 13:39:16.443467 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:16.443418 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-zqmlz" podStartSLOduration=2.033191424 podStartE2EDuration="4.44340063s" totalImagesPullingTime="2.410209206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:39:16.442407689 +0000 UTC m=+504.543220403" watchObservedRunningTime="2026-09-16 13:39:16.44340063 +0000 UTC m=+504.544213345" Sep 16 13:39:17.410432 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:17.410379 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-zqmlz" podUID="f8b40a8f-e3d9-4d60-bb3e-825fcbf324df" containerName="authorino" containerID="cri-o://00412ac98f314589d951b8b7918a18bcd59c79ad2c48b90d3a75e57b882c7dd9" gracePeriod=30 Sep 16 13:39:17.656981 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:17.656954 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-zqmlz" Sep 16 13:39:17.826449 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:17.826407 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnbwt\" (UniqueName: \"kubernetes.io/projected/f8b40a8f-e3d9-4d60-bb3e-825fcbf324df-kube-api-access-vnbwt\") pod \"f8b40a8f-e3d9-4d60-bb3e-825fcbf324df\" (UID: \"f8b40a8f-e3d9-4d60-bb3e-825fcbf324df\") " Sep 16 13:39:17.828810 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:17.828781 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b40a8f-e3d9-4d60-bb3e-825fcbf324df-kube-api-access-vnbwt" (OuterVolumeSpecName: "kube-api-access-vnbwt") pod "f8b40a8f-e3d9-4d60-bb3e-825fcbf324df" (UID: "f8b40a8f-e3d9-4d60-bb3e-825fcbf324df"). InnerVolumeSpecName "kube-api-access-vnbwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:39:17.927719 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:17.927686 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnbwt\" (UniqueName: \"kubernetes.io/projected/f8b40a8f-e3d9-4d60-bb3e-825fcbf324df-kube-api-access-vnbwt\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:39:18.415982 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:18.415942 2580 generic.go:356] "Generic (PLEG): container finished" podID="f8b40a8f-e3d9-4d60-bb3e-825fcbf324df" containerID="00412ac98f314589d951b8b7918a18bcd59c79ad2c48b90d3a75e57b882c7dd9" exitCode=0 Sep 16 13:39:18.415982 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:18.415983 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-zqmlz" Sep 16 13:39:18.416469 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:18.416028 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-zqmlz" event={"ID":"f8b40a8f-e3d9-4d60-bb3e-825fcbf324df","Type":"ContainerDied","Data":"00412ac98f314589d951b8b7918a18bcd59c79ad2c48b90d3a75e57b882c7dd9"} Sep 16 13:39:18.416469 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:18.416068 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-zqmlz" event={"ID":"f8b40a8f-e3d9-4d60-bb3e-825fcbf324df","Type":"ContainerDied","Data":"f0066a12a03fdf42dd6676fa627b33c851bfa06af1f49e540f7d97df58039996"} Sep 16 13:39:18.416469 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:18.416086 2580 scope.go:117] "RemoveContainer" containerID="00412ac98f314589d951b8b7918a18bcd59c79ad2c48b90d3a75e57b882c7dd9" Sep 16 13:39:18.424537 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:18.424518 2580 scope.go:117] "RemoveContainer" containerID="00412ac98f314589d951b8b7918a18bcd59c79ad2c48b90d3a75e57b882c7dd9" Sep 16 13:39:18.424865 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:39:18.424839 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00412ac98f314589d951b8b7918a18bcd59c79ad2c48b90d3a75e57b882c7dd9\": container with ID starting with 00412ac98f314589d951b8b7918a18bcd59c79ad2c48b90d3a75e57b882c7dd9 not found: ID does not exist" containerID="00412ac98f314589d951b8b7918a18bcd59c79ad2c48b90d3a75e57b882c7dd9" Sep 16 13:39:18.424945 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:18.424879 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00412ac98f314589d951b8b7918a18bcd59c79ad2c48b90d3a75e57b882c7dd9"} err="failed to get container status \"00412ac98f314589d951b8b7918a18bcd59c79ad2c48b90d3a75e57b882c7dd9\": rpc error: code = NotFound desc = could not find container \"00412ac98f314589d951b8b7918a18bcd59c79ad2c48b90d3a75e57b882c7dd9\": container with ID starting with 00412ac98f314589d951b8b7918a18bcd59c79ad2c48b90d3a75e57b882c7dd9 not found: ID does not exist" Sep 16 13:39:18.433597 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:18.433574 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zqmlz"] Sep 16 13:39:18.438920 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:18.438894 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zqmlz"] Sep 16 13:39:20.354460 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:20.354426 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8b40a8f-e3d9-4d60-bb3e-825fcbf324df" path="/var/lib/kubelet/pods/f8b40a8f-e3d9-4d60-bb3e-825fcbf324df/volumes" Sep 16 13:39:27.411762 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:27.411728 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-f8lqw" Sep 16 13:39:33.353863 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:33.353822 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:33.454650 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:33.454588 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:33.943284 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:33.943250 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:46.640845 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:46.640760 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:46.803623 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:46.803568 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:46.845591 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:46.845560 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-z274r"] Sep 16 13:39:46.845863 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:46.845851 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8b40a8f-e3d9-4d60-bb3e-825fcbf324df" containerName="authorino" Sep 16 13:39:46.845915 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:46.845865 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8b40a8f-e3d9-4d60-bb3e-825fcbf324df" containerName="authorino" Sep 16 13:39:46.845949 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:46.845926 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8b40a8f-e3d9-4d60-bb3e-825fcbf324df" containerName="authorino" Sep 16 13:39:46.847922 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:46.847902 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-z274r" Sep 16 13:39:46.850831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:46.850810 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xfzjj\"" Sep 16 13:39:46.856116 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:46.856096 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-z274r"] Sep 16 13:39:46.941643 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:46.941547 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-z274r"] Sep 16 13:39:46.941817 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:39:46.941799 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-29264], unattached volumes=[], failed to process volumes=[kube-api-access-29264]: context canceled" pod="kuadrant-system/authorino-557f876f69-z274r" podUID="6366ad9a-8ffa-497e-a9bb-465705387226" Sep 16 13:39:46.951548 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:46.951527 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29264\" (UniqueName: \"kubernetes.io/projected/6366ad9a-8ffa-497e-a9bb-465705387226-kube-api-access-29264\") pod \"authorino-557f876f69-z274r\" (UID: \"6366ad9a-8ffa-497e-a9bb-465705387226\") " pod="kuadrant-system/authorino-557f876f69-z274r" Sep 16 13:39:46.973030 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:46.973005 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-kwdg8"] Sep 16 13:39:46.975073 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:46.975055 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-kwdg8" Sep 16 13:39:46.977857 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:46.977835 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 16 13:39:46.987109 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:46.987087 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-kwdg8"] Sep 16 13:39:47.052355 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.052328 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bfada2ff-415e-412e-b7b4-eca80bfd3149-tls-cert\") pod \"authorino-6d4dfc649f-kwdg8\" (UID: \"bfada2ff-415e-412e-b7b4-eca80bfd3149\") " pod="kuadrant-system/authorino-6d4dfc649f-kwdg8" Sep 16 13:39:47.052513 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.052373 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29264\" (UniqueName: \"kubernetes.io/projected/6366ad9a-8ffa-497e-a9bb-465705387226-kube-api-access-29264\") pod \"authorino-557f876f69-z274r\" (UID: \"6366ad9a-8ffa-497e-a9bb-465705387226\") " pod="kuadrant-system/authorino-557f876f69-z274r" Sep 16 13:39:47.052513 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.052430 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch6zt\" (UniqueName: \"kubernetes.io/projected/bfada2ff-415e-412e-b7b4-eca80bfd3149-kube-api-access-ch6zt\") pod \"authorino-6d4dfc649f-kwdg8\" (UID: \"bfada2ff-415e-412e-b7b4-eca80bfd3149\") " pod="kuadrant-system/authorino-6d4dfc649f-kwdg8" Sep 16 13:39:47.064403 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.064373 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29264\" (UniqueName: \"kubernetes.io/projected/6366ad9a-8ffa-497e-a9bb-465705387226-kube-api-access-29264\") pod \"authorino-557f876f69-z274r\" (UID: \"6366ad9a-8ffa-497e-a9bb-465705387226\") " pod="kuadrant-system/authorino-557f876f69-z274r" Sep 16 13:39:47.089917 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.089890 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-kwdg8"] Sep 16 13:39:47.090144 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:39:47.090125 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-ch6zt tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-kwdg8" podUID="bfada2ff-415e-412e-b7b4-eca80bfd3149" Sep 16 13:39:47.127910 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.127867 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-684dfc4fdc-lqxxf"] Sep 16 13:39:47.130442 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.130424 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-684dfc4fdc-lqxxf" Sep 16 13:39:47.141833 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.141777 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-684dfc4fdc-lqxxf"] Sep 16 13:39:47.153025 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.152993 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch6zt\" (UniqueName: \"kubernetes.io/projected/bfada2ff-415e-412e-b7b4-eca80bfd3149-kube-api-access-ch6zt\") pod \"authorino-6d4dfc649f-kwdg8\" (UID: \"bfada2ff-415e-412e-b7b4-eca80bfd3149\") " pod="kuadrant-system/authorino-6d4dfc649f-kwdg8" Sep 16 13:39:47.153165 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.153033 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/08eebbd0-41ee-4952-9b0b-bacca0bb1157-tls-cert\") pod \"authorino-684dfc4fdc-lqxxf\" (UID: \"08eebbd0-41ee-4952-9b0b-bacca0bb1157\") " pod="kuadrant-system/authorino-684dfc4fdc-lqxxf" Sep 16 13:39:47.153165 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.153051 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlbh8\" (UniqueName: \"kubernetes.io/projected/08eebbd0-41ee-4952-9b0b-bacca0bb1157-kube-api-access-mlbh8\") pod \"authorino-684dfc4fdc-lqxxf\" (UID: \"08eebbd0-41ee-4952-9b0b-bacca0bb1157\") " pod="kuadrant-system/authorino-684dfc4fdc-lqxxf" Sep 16 13:39:47.153337 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.153163 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bfada2ff-415e-412e-b7b4-eca80bfd3149-tls-cert\") pod \"authorino-6d4dfc649f-kwdg8\" (UID: \"bfada2ff-415e-412e-b7b4-eca80bfd3149\") " pod="kuadrant-system/authorino-6d4dfc649f-kwdg8" Sep 16 13:39:47.155843 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.155816 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bfada2ff-415e-412e-b7b4-eca80bfd3149-tls-cert\") pod \"authorino-6d4dfc649f-kwdg8\" (UID: \"bfada2ff-415e-412e-b7b4-eca80bfd3149\") " pod="kuadrant-system/authorino-6d4dfc649f-kwdg8" Sep 16 13:39:47.163050 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.163031 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch6zt\" (UniqueName: \"kubernetes.io/projected/bfada2ff-415e-412e-b7b4-eca80bfd3149-kube-api-access-ch6zt\") pod \"authorino-6d4dfc649f-kwdg8\" (UID: \"bfada2ff-415e-412e-b7b4-eca80bfd3149\") " pod="kuadrant-system/authorino-6d4dfc649f-kwdg8" Sep 16 13:39:47.253765 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.253734 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/08eebbd0-41ee-4952-9b0b-bacca0bb1157-tls-cert\") pod \"authorino-684dfc4fdc-lqxxf\" (UID: \"08eebbd0-41ee-4952-9b0b-bacca0bb1157\") " pod="kuadrant-system/authorino-684dfc4fdc-lqxxf" Sep 16 13:39:47.253944 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.253773 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlbh8\" (UniqueName: \"kubernetes.io/projected/08eebbd0-41ee-4952-9b0b-bacca0bb1157-kube-api-access-mlbh8\") pod \"authorino-684dfc4fdc-lqxxf\" (UID: \"08eebbd0-41ee-4952-9b0b-bacca0bb1157\") " pod="kuadrant-system/authorino-684dfc4fdc-lqxxf" Sep 16 13:39:47.256368 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.256345 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/08eebbd0-41ee-4952-9b0b-bacca0bb1157-tls-cert\") pod \"authorino-684dfc4fdc-lqxxf\" (UID: \"08eebbd0-41ee-4952-9b0b-bacca0bb1157\") " pod="kuadrant-system/authorino-684dfc4fdc-lqxxf" Sep 16 13:39:47.262497 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.262470 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlbh8\" (UniqueName: \"kubernetes.io/projected/08eebbd0-41ee-4952-9b0b-bacca0bb1157-kube-api-access-mlbh8\") pod \"authorino-684dfc4fdc-lqxxf\" (UID: \"08eebbd0-41ee-4952-9b0b-bacca0bb1157\") " pod="kuadrant-system/authorino-684dfc4fdc-lqxxf" Sep 16 13:39:47.438941 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.438901 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-684dfc4fdc-lqxxf" Sep 16 13:39:47.533103 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.533033 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-z274r" Sep 16 13:39:47.533245 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.533032 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-kwdg8" Sep 16 13:39:47.539391 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.539366 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-z274r" Sep 16 13:39:47.542277 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.542259 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-kwdg8" Sep 16 13:39:47.556098 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.556079 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29264\" (UniqueName: \"kubernetes.io/projected/6366ad9a-8ffa-497e-a9bb-465705387226-kube-api-access-29264\") pod \"6366ad9a-8ffa-497e-a9bb-465705387226\" (UID: \"6366ad9a-8ffa-497e-a9bb-465705387226\") " Sep 16 13:39:47.556201 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.556103 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bfada2ff-415e-412e-b7b4-eca80bfd3149-tls-cert\") pod \"bfada2ff-415e-412e-b7b4-eca80bfd3149\" (UID: \"bfada2ff-415e-412e-b7b4-eca80bfd3149\") " Sep 16 13:39:47.556201 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.556132 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch6zt\" (UniqueName: \"kubernetes.io/projected/bfada2ff-415e-412e-b7b4-eca80bfd3149-kube-api-access-ch6zt\") pod \"bfada2ff-415e-412e-b7b4-eca80bfd3149\" (UID: \"bfada2ff-415e-412e-b7b4-eca80bfd3149\") " Sep 16 13:39:47.558367 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.558337 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6366ad9a-8ffa-497e-a9bb-465705387226-kube-api-access-29264" (OuterVolumeSpecName: "kube-api-access-29264") pod "6366ad9a-8ffa-497e-a9bb-465705387226" (UID: "6366ad9a-8ffa-497e-a9bb-465705387226"). InnerVolumeSpecName "kube-api-access-29264". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:39:47.558367 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.558356 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfada2ff-415e-412e-b7b4-eca80bfd3149-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "bfada2ff-415e-412e-b7b4-eca80bfd3149" (UID: "bfada2ff-415e-412e-b7b4-eca80bfd3149"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:39:47.558491 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.558378 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfada2ff-415e-412e-b7b4-eca80bfd3149-kube-api-access-ch6zt" (OuterVolumeSpecName: "kube-api-access-ch6zt") pod "bfada2ff-415e-412e-b7b4-eca80bfd3149" (UID: "bfada2ff-415e-412e-b7b4-eca80bfd3149"). InnerVolumeSpecName "kube-api-access-ch6zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:39:47.657578 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.657543 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch6zt\" (UniqueName: \"kubernetes.io/projected/bfada2ff-415e-412e-b7b4-eca80bfd3149-kube-api-access-ch6zt\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:39:47.657578 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.657572 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29264\" (UniqueName: \"kubernetes.io/projected/6366ad9a-8ffa-497e-a9bb-465705387226-kube-api-access-29264\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:39:47.657578 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.657582 2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bfada2ff-415e-412e-b7b4-eca80bfd3149-tls-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:39:47.712378 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:47.712355 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-684dfc4fdc-lqxxf"] Sep 16 13:39:47.714880 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:39:47.714852 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08eebbd0_41ee_4952_9b0b_bacca0bb1157.slice/crio-88334ffa6e5df3607e0bf8680e4d1b564df59c1dd32fb11d26a329aafdd3420e WatchSource:0}: Error finding container 88334ffa6e5df3607e0bf8680e4d1b564df59c1dd32fb11d26a329aafdd3420e: Status 404 returned error can't find the container with id 88334ffa6e5df3607e0bf8680e4d1b564df59c1dd32fb11d26a329aafdd3420e Sep 16 13:39:48.539005 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:48.538967 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-684dfc4fdc-lqxxf" event={"ID":"08eebbd0-41ee-4952-9b0b-bacca0bb1157","Type":"ContainerStarted","Data":"999942067d841a59870c41d7837e5068eb2027ae5b7cac00c63a346e57a14ab8"} Sep 16 13:39:48.539005 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:48.538992 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-z274r" Sep 16 13:39:48.539005 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:48.539016 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-684dfc4fdc-lqxxf" event={"ID":"08eebbd0-41ee-4952-9b0b-bacca0bb1157","Type":"ContainerStarted","Data":"88334ffa6e5df3607e0bf8680e4d1b564df59c1dd32fb11d26a329aafdd3420e"} Sep 16 13:39:48.539296 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:48.538984 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-kwdg8" Sep 16 13:39:48.558177 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:48.558121 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-684dfc4fdc-lqxxf" podStartSLOduration=0.962965664 podStartE2EDuration="1.55810491s" totalImagesPullingTime="595.139246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:39:48.555949412 +0000 UTC m=+536.656762140" watchObservedRunningTime="2026-09-16 13:39:48.55810491 +0000 UTC m=+536.658917624" Sep 16 13:39:48.584826 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:48.584738 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-kwdg8"] Sep 16 13:39:48.601107 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:48.601071 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-kwdg8"] Sep 16 13:39:48.630041 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:48.630003 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-z274r"] Sep 16 13:39:48.632407 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:48.632381 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-z274r"] Sep 16 13:39:50.354772 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:50.354736 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6366ad9a-8ffa-497e-a9bb-465705387226" path="/var/lib/kubelet/pods/6366ad9a-8ffa-497e-a9bb-465705387226/volumes" Sep 16 13:39:50.355140 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:50.354986 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfada2ff-415e-412e-b7b4-eca80bfd3149" path="/var/lib/kubelet/pods/bfada2ff-415e-412e-b7b4-eca80bfd3149/volumes" Sep 16 13:39:53.742897 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:53.742862 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:53.822707 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:53.822680 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:53.884962 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:53.884931 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:54.080078 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:54.080002 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:54.180762 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:54.180730 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:57.448794 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:57.448758 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:57.540273 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:39:57.540238 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:40:17.343658 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:40:17.343590 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:40:18.291819 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:40:18.291786 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:40:18.393085 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:40:18.393048 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:40:28.982889 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:40:28.982853 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:40:29.075666 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:40:29.075592 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:40:30.567083 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:40:30.567051 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:40:30.664442 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:40:30.664406 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:40:30.805088 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:40:30.805059 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:40:31.273144 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:40:31.273110 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:40:31.422094 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:40:31.422061 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:40:31.690565 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:40:31.690477 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:40:41.420723 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:40:41.420687 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:40:41.521668 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:40:41.521641 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:40:51.037268 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:40:51.037210 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:40:51.605942 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:40:51.605907 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:40:57.842065 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:40:57.842034 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:40:58.644550 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:40:58.644517 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:41:04.306640 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:04.306587 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:41:05.312654 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:05.312594 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:41:13.185690 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.185657 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql"] Sep 16 13:41:13.187875 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.187854 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:13.190906 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.190881 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 16 13:41:13.192069 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.192043 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 16 13:41:13.192188 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.192100 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-m5ll2\"" Sep 16 13:41:13.192188 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.192100 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 16 13:41:13.200519 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.200494 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql"] Sep 16 13:41:13.217012 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.216984 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:41:13.292026 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.291994 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/097529a4-8834-45e1-a5f5-05f80f8157de-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xpgql\" (UID: \"097529a4-8834-45e1-a5f5-05f80f8157de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:13.292026 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.292029 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/097529a4-8834-45e1-a5f5-05f80f8157de-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xpgql\" (UID: \"097529a4-8834-45e1-a5f5-05f80f8157de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:13.292218 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.292050 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhfp8\" (UniqueName: \"kubernetes.io/projected/097529a4-8834-45e1-a5f5-05f80f8157de-kube-api-access-lhfp8\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xpgql\" (UID: \"097529a4-8834-45e1-a5f5-05f80f8157de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:13.292218 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.292078 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/097529a4-8834-45e1-a5f5-05f80f8157de-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xpgql\" (UID: \"097529a4-8834-45e1-a5f5-05f80f8157de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:13.292218 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.292160 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/097529a4-8834-45e1-a5f5-05f80f8157de-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xpgql\" (UID: \"097529a4-8834-45e1-a5f5-05f80f8157de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:13.292218 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.292206 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/097529a4-8834-45e1-a5f5-05f80f8157de-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xpgql\" (UID: \"097529a4-8834-45e1-a5f5-05f80f8157de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:13.393018 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.392946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/097529a4-8834-45e1-a5f5-05f80f8157de-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xpgql\" (UID: \"097529a4-8834-45e1-a5f5-05f80f8157de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:13.393018 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.392983 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/097529a4-8834-45e1-a5f5-05f80f8157de-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xpgql\" (UID: \"097529a4-8834-45e1-a5f5-05f80f8157de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:13.393018 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.393002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhfp8\" (UniqueName: \"kubernetes.io/projected/097529a4-8834-45e1-a5f5-05f80f8157de-kube-api-access-lhfp8\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xpgql\" (UID: \"097529a4-8834-45e1-a5f5-05f80f8157de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:13.393250 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.393028 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/097529a4-8834-45e1-a5f5-05f80f8157de-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xpgql\" (UID: \"097529a4-8834-45e1-a5f5-05f80f8157de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:13.393250 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.393081 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/097529a4-8834-45e1-a5f5-05f80f8157de-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xpgql\" (UID: \"097529a4-8834-45e1-a5f5-05f80f8157de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:13.393250 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.393128 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/097529a4-8834-45e1-a5f5-05f80f8157de-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xpgql\" (UID: \"097529a4-8834-45e1-a5f5-05f80f8157de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:13.393481 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.393460 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/097529a4-8834-45e1-a5f5-05f80f8157de-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xpgql\" (UID: \"097529a4-8834-45e1-a5f5-05f80f8157de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:13.393536 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.393484 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/097529a4-8834-45e1-a5f5-05f80f8157de-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xpgql\" (UID: \"097529a4-8834-45e1-a5f5-05f80f8157de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:13.393536 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.393514 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/097529a4-8834-45e1-a5f5-05f80f8157de-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xpgql\" (UID: \"097529a4-8834-45e1-a5f5-05f80f8157de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:13.395464 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.395442 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/097529a4-8834-45e1-a5f5-05f80f8157de-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xpgql\" (UID: \"097529a4-8834-45e1-a5f5-05f80f8157de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:13.395727 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.395710 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/097529a4-8834-45e1-a5f5-05f80f8157de-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xpgql\" (UID: \"097529a4-8834-45e1-a5f5-05f80f8157de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:13.401846 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.401812 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhfp8\" (UniqueName: \"kubernetes.io/projected/097529a4-8834-45e1-a5f5-05f80f8157de-kube-api-access-lhfp8\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xpgql\" (UID: \"097529a4-8834-45e1-a5f5-05f80f8157de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:13.499454 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.499422 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:13.777450 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.777370 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql"] Sep 16 13:41:13.780208 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:41:13.780181 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod097529a4_8834_45e1_a5f5_05f80f8157de.slice/crio-5388fed5d05516fb9e9f8b7050ff3b54795b54f02d287cd96c9d226c7794ecef WatchSource:0}: Error finding container 5388fed5d05516fb9e9f8b7050ff3b54795b54f02d287cd96c9d226c7794ecef: Status 404 returned error can't find the container with id 5388fed5d05516fb9e9f8b7050ff3b54795b54f02d287cd96c9d226c7794ecef Sep 16 13:41:13.896780 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:13.896747 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" event={"ID":"097529a4-8834-45e1-a5f5-05f80f8157de","Type":"ContainerStarted","Data":"5388fed5d05516fb9e9f8b7050ff3b54795b54f02d287cd96c9d226c7794ecef"} Sep 16 13:41:14.417766 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:14.417728 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:41:15.907508 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:15.907471 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" event={"ID":"097529a4-8834-45e1-a5f5-05f80f8157de","Type":"ContainerStarted","Data":"eb402914a7cd9902a4510ca9baa8401d9961ba785b31236abf6f1dd81c473bcf"} Sep 16 13:41:15.928248 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:15.928201 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" podStartSLOduration=1.7685858639999998 podStartE2EDuration="2.928185318s" totalImagesPullingTime="1.159599454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:41:15.92692695 +0000 UTC m=+624.027739694" watchObservedRunningTime="2026-09-16 13:41:15.928185318 +0000 UTC m=+624.028998037" Sep 16 13:41:23.500193 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:23.500136 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:23.500719 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:23.500229 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:23.512777 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:23.512754 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:23.952174 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:23.952102 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xpgql" Sep 16 13:41:25.703370 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:25.703309 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:41:30.980537 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:30.980510 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn"] Sep 16 13:41:30.982845 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:30.982827 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:30.985993 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:30.985968 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 16 13:41:31.010788 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.010754 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn"] Sep 16 13:41:31.039162 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.039130 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759b88a0-2f0d-4e7f-bb7c-efe9e7efe470-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn\" (UID: \"759b88a0-2f0d-4e7f-bb7c-efe9e7efe470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:31.039162 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.039164 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/759b88a0-2f0d-4e7f-bb7c-efe9e7efe470-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn\" (UID: \"759b88a0-2f0d-4e7f-bb7c-efe9e7efe470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:31.039361 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.039212 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5bkq\" (UniqueName: \"kubernetes.io/projected/759b88a0-2f0d-4e7f-bb7c-efe9e7efe470-kube-api-access-b5bkq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn\" (UID: \"759b88a0-2f0d-4e7f-bb7c-efe9e7efe470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:31.039361 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.039267 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/759b88a0-2f0d-4e7f-bb7c-efe9e7efe470-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn\" (UID: \"759b88a0-2f0d-4e7f-bb7c-efe9e7efe470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:31.039361 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.039289 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/759b88a0-2f0d-4e7f-bb7c-efe9e7efe470-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn\" (UID: \"759b88a0-2f0d-4e7f-bb7c-efe9e7efe470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:31.039361 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.039319 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/759b88a0-2f0d-4e7f-bb7c-efe9e7efe470-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn\" (UID: \"759b88a0-2f0d-4e7f-bb7c-efe9e7efe470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:31.040182 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.040161 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:41:31.140033 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.139998 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/759b88a0-2f0d-4e7f-bb7c-efe9e7efe470-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn\" (UID: \"759b88a0-2f0d-4e7f-bb7c-efe9e7efe470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:31.140233 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.140065 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759b88a0-2f0d-4e7f-bb7c-efe9e7efe470-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn\" (UID: \"759b88a0-2f0d-4e7f-bb7c-efe9e7efe470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:31.140233 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.140098 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/759b88a0-2f0d-4e7f-bb7c-efe9e7efe470-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn\" (UID: \"759b88a0-2f0d-4e7f-bb7c-efe9e7efe470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:31.140233 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.140148 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5bkq\" (UniqueName: \"kubernetes.io/projected/759b88a0-2f0d-4e7f-bb7c-efe9e7efe470-kube-api-access-b5bkq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn\" (UID: \"759b88a0-2f0d-4e7f-bb7c-efe9e7efe470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:31.140233 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.140186 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/759b88a0-2f0d-4e7f-bb7c-efe9e7efe470-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn\" (UID: \"759b88a0-2f0d-4e7f-bb7c-efe9e7efe470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:31.140233 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.140218 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/759b88a0-2f0d-4e7f-bb7c-efe9e7efe470-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn\" (UID: \"759b88a0-2f0d-4e7f-bb7c-efe9e7efe470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:31.140557 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.140407 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759b88a0-2f0d-4e7f-bb7c-efe9e7efe470-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn\" (UID: \"759b88a0-2f0d-4e7f-bb7c-efe9e7efe470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:31.140557 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.140472 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/759b88a0-2f0d-4e7f-bb7c-efe9e7efe470-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn\" (UID: \"759b88a0-2f0d-4e7f-bb7c-efe9e7efe470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:31.140677 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.140649 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/759b88a0-2f0d-4e7f-bb7c-efe9e7efe470-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn\" (UID: \"759b88a0-2f0d-4e7f-bb7c-efe9e7efe470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:31.142662 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.142640 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/759b88a0-2f0d-4e7f-bb7c-efe9e7efe470-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn\" (UID: \"759b88a0-2f0d-4e7f-bb7c-efe9e7efe470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:31.142839 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.142823 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/759b88a0-2f0d-4e7f-bb7c-efe9e7efe470-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn\" (UID: \"759b88a0-2f0d-4e7f-bb7c-efe9e7efe470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:31.157283 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.157256 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5bkq\" (UniqueName: \"kubernetes.io/projected/759b88a0-2f0d-4e7f-bb7c-efe9e7efe470-kube-api-access-b5bkq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn\" (UID: \"759b88a0-2f0d-4e7f-bb7c-efe9e7efe470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:31.294834 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.294748 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:31.570560 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.570535 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn"] Sep 16 13:41:31.572410 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:41:31.572384 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod759b88a0_2f0d_4e7f_bb7c_efe9e7efe470.slice/crio-fdb0f2a3d43aadfc1c7df62cea1fcb7c0385b412b846b29835d84374b09c2619 WatchSource:0}: Error finding container fdb0f2a3d43aadfc1c7df62cea1fcb7c0385b412b846b29835d84374b09c2619: Status 404 returned error can't find the container with id fdb0f2a3d43aadfc1c7df62cea1fcb7c0385b412b846b29835d84374b09c2619 Sep 16 13:41:31.573818 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.573798 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 13:41:31.982799 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.982755 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" event={"ID":"759b88a0-2f0d-4e7f-bb7c-efe9e7efe470","Type":"ContainerStarted","Data":"5cb6522930203c72afa5f6f839f965e516b24e19e92fb06b54b31ce2994bb81c"} Sep 16 13:41:31.983255 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:31.982841 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" event={"ID":"759b88a0-2f0d-4e7f-bb7c-efe9e7efe470","Type":"ContainerStarted","Data":"fdb0f2a3d43aadfc1c7df62cea1fcb7c0385b412b846b29835d84374b09c2619"} Sep 16 13:41:32.004743 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:32.004697 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" podStartSLOduration=1.821973439 podStartE2EDuration="2.004684354s" totalImagesPullingTime="182.710915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:41:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:41:32.00358355 +0000 UTC m=+640.104396266" watchObservedRunningTime="2026-09-16 13:41:32.004684354 +0000 UTC m=+640.105497069" Sep 16 13:41:32.409936 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:32.409824 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:41:38.320919 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:38.320881 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:41:40.135880 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:40.135848 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:41:41.295704 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:41.295665 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:41.296115 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:41.295729 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:41.308124 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:41.308096 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:42.044664 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:42.044635 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn" Sep 16 13:41:45.485229 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.485201 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9"] Sep 16 13:41:45.491769 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.491746 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:45.494522 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.494497 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 16 13:41:45.499921 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.499898 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9"] Sep 16 13:41:45.527056 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.527024 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:41:45.568211 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.568179 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15242a69-7482-4d28-9e00-388900df6334-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9\" (UID: \"15242a69-7482-4d28-9e00-388900df6334\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:45.568211 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.568212 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15242a69-7482-4d28-9e00-388900df6334-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9\" (UID: \"15242a69-7482-4d28-9e00-388900df6334\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:45.568384 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.568237 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15242a69-7482-4d28-9e00-388900df6334-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9\" (UID: \"15242a69-7482-4d28-9e00-388900df6334\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:45.568384 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.568309 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwcj9\" (UniqueName: \"kubernetes.io/projected/15242a69-7482-4d28-9e00-388900df6334-kube-api-access-nwcj9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9\" (UID: \"15242a69-7482-4d28-9e00-388900df6334\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:45.568384 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.568351 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15242a69-7482-4d28-9e00-388900df6334-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9\" (UID: \"15242a69-7482-4d28-9e00-388900df6334\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:45.568496 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.568398 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15242a69-7482-4d28-9e00-388900df6334-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9\" (UID: \"15242a69-7482-4d28-9e00-388900df6334\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:45.669142 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.669104 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15242a69-7482-4d28-9e00-388900df6334-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9\" (UID: \"15242a69-7482-4d28-9e00-388900df6334\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:45.669142 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.669144 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15242a69-7482-4d28-9e00-388900df6334-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9\" (UID: \"15242a69-7482-4d28-9e00-388900df6334\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:45.669358 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.669167 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15242a69-7482-4d28-9e00-388900df6334-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9\" (UID: \"15242a69-7482-4d28-9e00-388900df6334\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:45.669358 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.669196 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwcj9\" (UniqueName: \"kubernetes.io/projected/15242a69-7482-4d28-9e00-388900df6334-kube-api-access-nwcj9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9\" (UID: \"15242a69-7482-4d28-9e00-388900df6334\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:45.669358 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.669233 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15242a69-7482-4d28-9e00-388900df6334-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9\" (UID: \"15242a69-7482-4d28-9e00-388900df6334\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:45.669358 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.669289 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15242a69-7482-4d28-9e00-388900df6334-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9\" (UID: \"15242a69-7482-4d28-9e00-388900df6334\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:45.669707 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.669671 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15242a69-7482-4d28-9e00-388900df6334-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9\" (UID: \"15242a69-7482-4d28-9e00-388900df6334\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:45.669707 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.669697 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15242a69-7482-4d28-9e00-388900df6334-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9\" (UID: \"15242a69-7482-4d28-9e00-388900df6334\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:45.669860 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.669713 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15242a69-7482-4d28-9e00-388900df6334-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9\" (UID: \"15242a69-7482-4d28-9e00-388900df6334\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:45.671709 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.671689 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15242a69-7482-4d28-9e00-388900df6334-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9\" (UID: \"15242a69-7482-4d28-9e00-388900df6334\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:45.671841 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.671823 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15242a69-7482-4d28-9e00-388900df6334-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9\" (UID: \"15242a69-7482-4d28-9e00-388900df6334\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:45.683420 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.683398 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwcj9\" (UniqueName: \"kubernetes.io/projected/15242a69-7482-4d28-9e00-388900df6334-kube-api-access-nwcj9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9\" (UID: \"15242a69-7482-4d28-9e00-388900df6334\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:45.801839 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:45.801749 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:46.074061 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:46.074023 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9"] Sep 16 13:41:46.076234 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:41:46.076204 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15242a69_7482_4d28_9e00_388900df6334.slice/crio-243f0e1fd1c02a87fdef78cfcbffa8598fefc86f2f126b4e97ed5ec69b3aac6a WatchSource:0}: Error finding container 243f0e1fd1c02a87fdef78cfcbffa8598fefc86f2f126b4e97ed5ec69b3aac6a: Status 404 returned error can't find the container with id 243f0e1fd1c02a87fdef78cfcbffa8598fefc86f2f126b4e97ed5ec69b3aac6a Sep 16 13:41:47.058018 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:47.057981 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" event={"ID":"15242a69-7482-4d28-9e00-388900df6334","Type":"ContainerStarted","Data":"baaba8333e3a2a4cdfed84bf111e2763c48e6cf4ec42a50954ada5ca937b9e22"} Sep 16 13:41:47.058497 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:47.058032 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" event={"ID":"15242a69-7482-4d28-9e00-388900df6334","Type":"ContainerStarted","Data":"243f0e1fd1c02a87fdef78cfcbffa8598fefc86f2f126b4e97ed5ec69b3aac6a"} Sep 16 13:41:47.083013 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:47.082961 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" podStartSLOduration=1.881908271 podStartE2EDuration="2.082939482s" totalImagesPullingTime="201.031211ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:41:47.082428884 +0000 UTC m=+655.183241598" watchObservedRunningTime="2026-09-16 13:41:47.082939482 +0000 UTC m=+655.183752198" Sep 16 13:41:48.005424 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:48.005386 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:41:55.802371 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:55.802336 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:55.802992 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:55.802391 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:55.815266 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:55.815243 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:41:56.111971 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:41:56.111887 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9" Sep 16 13:42:04.118063 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:42:04.118030 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:14.269323 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:42:14.269250 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:51.349258 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:42:51.349176 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:51.414157 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:42:51.414110 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:56.373421 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:42:56.373360 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:56.478519 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:42:56.478480 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:17.331068 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:17.331036 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-684dfc4fdc-lqxxf"] Sep 16 13:43:17.331562 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:17.331215 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-684dfc4fdc-lqxxf" podUID="08eebbd0-41ee-4952-9b0b-bacca0bb1157" containerName="authorino" containerID="cri-o://999942067d841a59870c41d7837e5068eb2027ae5b7cac00c63a346e57a14ab8" gracePeriod=30 Sep 16 13:43:17.456412 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:17.456380 2580 generic.go:356] "Generic (PLEG): container finished" podID="08eebbd0-41ee-4952-9b0b-bacca0bb1157" containerID="999942067d841a59870c41d7837e5068eb2027ae5b7cac00c63a346e57a14ab8" exitCode=0 Sep 16 13:43:17.456545 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:17.456449 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-684dfc4fdc-lqxxf" event={"ID":"08eebbd0-41ee-4952-9b0b-bacca0bb1157","Type":"ContainerDied","Data":"999942067d841a59870c41d7837e5068eb2027ae5b7cac00c63a346e57a14ab8"} Sep 16 13:43:17.566355 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:17.566329 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-684dfc4fdc-lqxxf" Sep 16 13:43:17.669974 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:17.669885 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlbh8\" (UniqueName: \"kubernetes.io/projected/08eebbd0-41ee-4952-9b0b-bacca0bb1157-kube-api-access-mlbh8\") pod \"08eebbd0-41ee-4952-9b0b-bacca0bb1157\" (UID: \"08eebbd0-41ee-4952-9b0b-bacca0bb1157\") " Sep 16 13:43:17.669974 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:17.669938 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/08eebbd0-41ee-4952-9b0b-bacca0bb1157-tls-cert\") pod \"08eebbd0-41ee-4952-9b0b-bacca0bb1157\" (UID: \"08eebbd0-41ee-4952-9b0b-bacca0bb1157\") " Sep 16 13:43:17.672304 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:17.672275 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08eebbd0-41ee-4952-9b0b-bacca0bb1157-kube-api-access-mlbh8" (OuterVolumeSpecName: "kube-api-access-mlbh8") pod "08eebbd0-41ee-4952-9b0b-bacca0bb1157" (UID: "08eebbd0-41ee-4952-9b0b-bacca0bb1157"). InnerVolumeSpecName "kube-api-access-mlbh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:43:17.680752 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:17.680715 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08eebbd0-41ee-4952-9b0b-bacca0bb1157-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "08eebbd0-41ee-4952-9b0b-bacca0bb1157" (UID: "08eebbd0-41ee-4952-9b0b-bacca0bb1157"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:43:17.771468 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:17.771431 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlbh8\" (UniqueName: \"kubernetes.io/projected/08eebbd0-41ee-4952-9b0b-bacca0bb1157-kube-api-access-mlbh8\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:43:17.771468 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:17.771460 2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/08eebbd0-41ee-4952-9b0b-bacca0bb1157-tls-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:43:17.970551 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:17.970523 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:18.090945 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:18.090915 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:18.461400 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:18.461371 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-684dfc4fdc-lqxxf" Sep 16 13:43:18.461901 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:18.461412 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-684dfc4fdc-lqxxf" event={"ID":"08eebbd0-41ee-4952-9b0b-bacca0bb1157","Type":"ContainerDied","Data":"88334ffa6e5df3607e0bf8680e4d1b564df59c1dd32fb11d26a329aafdd3420e"} Sep 16 13:43:18.461901 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:18.461456 2580 scope.go:117] "RemoveContainer" containerID="999942067d841a59870c41d7837e5068eb2027ae5b7cac00c63a346e57a14ab8" Sep 16 13:43:18.483433 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:18.483403 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-684dfc4fdc-lqxxf"] Sep 16 13:43:18.489484 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:18.489458 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-684dfc4fdc-lqxxf"] Sep 16 13:43:19.106146 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:19.106117 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:19.209552 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:19.209516 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:20.354182 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:20.354148 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08eebbd0-41ee-4952-9b0b-bacca0bb1157" path="/var/lib/kubelet/pods/08eebbd0-41ee-4952-9b0b-bacca0bb1157/volumes" Sep 16 13:43:46.629230 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:46.629193 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:47.018359 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:47.018325 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:47.326845 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:47.326768 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:47.701405 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:47.701317 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:48.203775 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:48.203743 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:48.767843 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:48.767810 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:49.009064 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:49.009029 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:49.226732 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:49.226705 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7321-619a5e"] Sep 16 13:43:49.227012 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:49.227000 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08eebbd0-41ee-4952-9b0b-bacca0bb1157" containerName="authorino" Sep 16 13:43:49.227055 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:49.227015 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="08eebbd0-41ee-4952-9b0b-bacca0bb1157" containerName="authorino" Sep 16 13:43:49.227092 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:49.227082 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="08eebbd0-41ee-4952-9b0b-bacca0bb1157" containerName="authorino" Sep 16 13:43:49.230044 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:49.230023 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-619a5e" Sep 16 13:43:49.236882 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:49.236861 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2kqls\"" Sep 16 13:43:49.262409 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:49.262385 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7321-619a5e"] Sep 16 13:43:49.326282 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:49.326252 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:49.340496 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:49.340466 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sttj4\" (UniqueName: \"kubernetes.io/projected/74d35d0b-69ce-40f0-9fd6-c40f3efb0add-kube-api-access-sttj4\") pod \"test-curl-7321-619a5e\" (UID: \"74d35d0b-69ce-40f0-9fd6-c40f3efb0add\") " pod="opendatahub/test-curl-7321-619a5e" Sep 16 13:43:49.441427 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:49.441392 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sttj4\" (UniqueName: \"kubernetes.io/projected/74d35d0b-69ce-40f0-9fd6-c40f3efb0add-kube-api-access-sttj4\") pod \"test-curl-7321-619a5e\" (UID: \"74d35d0b-69ce-40f0-9fd6-c40f3efb0add\") " pod="opendatahub/test-curl-7321-619a5e" Sep 16 13:43:49.452878 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:49.452855 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sttj4\" (UniqueName: \"kubernetes.io/projected/74d35d0b-69ce-40f0-9fd6-c40f3efb0add-kube-api-access-sttj4\") pod \"test-curl-7321-619a5e\" (UID: \"74d35d0b-69ce-40f0-9fd6-c40f3efb0add\") " pod="opendatahub/test-curl-7321-619a5e" Sep 16 13:43:49.538914 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:49.538835 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-619a5e" Sep 16 13:43:49.843444 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:49.843419 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7321-619a5e"] Sep 16 13:43:49.846155 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:43:49.846125 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74d35d0b_69ce_40f0_9fd6_c40f3efb0add.slice/crio-cee670174cebbbf4e5031cc79325d955a6243522c69476ec57b7411937bb6a0d WatchSource:0}: Error finding container cee670174cebbbf4e5031cc79325d955a6243522c69476ec57b7411937bb6a0d: Status 404 returned error can't find the container with id cee670174cebbbf4e5031cc79325d955a6243522c69476ec57b7411937bb6a0d Sep 16 13:43:50.612867 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:50.612833 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7321-619a5e" event={"ID":"74d35d0b-69ce-40f0-9fd6-c40f3efb0add","Type":"ContainerStarted","Data":"cee670174cebbbf4e5031cc79325d955a6243522c69476ec57b7411937bb6a0d"} Sep 16 13:43:51.069332 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:51.069296 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:51.341387 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:51.341304 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:51.895122 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:51.895089 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:52.149178 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:52.149155 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:43:52.165024 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:52.164990 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:52.622472 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:52.622434 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7321-619a5e" event={"ID":"74d35d0b-69ce-40f0-9fd6-c40f3efb0add","Type":"ContainerStarted","Data":"bb3a70798466958cec549dc684442e6ea32105c1e66bdc556ba5a138a41a797c"} Sep 16 13:43:52.653935 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:52.653888 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7321-619a5e" podStartSLOduration=1.355224485 podStartE2EDuration="3.653872102s" totalImagesPullingTime="2.298647617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:52.650941061 +0000 UTC m=+780.751753777" watchObservedRunningTime="2026-09-16 13:43:52.653872102 +0000 UTC m=+780.754684816" Sep 16 13:43:53.902844 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:53.902811 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7321-619a5e"] Sep 16 13:43:53.906064 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:53.906039 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7321-619a5e"] Sep 16 13:43:54.143934 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:54.143899 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7321-31dd25"] Sep 16 13:43:54.144257 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:54.144244 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74d35d0b-69ce-40f0-9fd6-c40f3efb0add" containerName="test-curl-7321-619a5e" Sep 16 13:43:54.144301 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:54.144259 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="74d35d0b-69ce-40f0-9fd6-c40f3efb0add" containerName="test-curl-7321-619a5e" Sep 16 13:43:54.144335 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:54.144319 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="74d35d0b-69ce-40f0-9fd6-c40f3efb0add" containerName="test-curl-7321-619a5e" Sep 16 13:43:54.148473 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:54.148458 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-31dd25" Sep 16 13:43:54.162565 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:54.162542 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:54.179224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:54.179201 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7321-31dd25"] Sep 16 13:43:54.183038 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:54.183017 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgltd\" (UniqueName: \"kubernetes.io/projected/8c243a10-9a39-400c-8df2-896ab7a1b6d4-kube-api-access-mgltd\") pod \"test-curl-7321-31dd25\" (UID: \"8c243a10-9a39-400c-8df2-896ab7a1b6d4\") " pod="opendatahub/test-curl-7321-31dd25" Sep 16 13:43:54.284379 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:54.284347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgltd\" (UniqueName: \"kubernetes.io/projected/8c243a10-9a39-400c-8df2-896ab7a1b6d4-kube-api-access-mgltd\") pod \"test-curl-7321-31dd25\" (UID: \"8c243a10-9a39-400c-8df2-896ab7a1b6d4\") " pod="opendatahub/test-curl-7321-31dd25" Sep 16 13:43:54.312922 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:54.312885 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgltd\" (UniqueName: \"kubernetes.io/projected/8c243a10-9a39-400c-8df2-896ab7a1b6d4-kube-api-access-mgltd\") pod \"test-curl-7321-31dd25\" (UID: \"8c243a10-9a39-400c-8df2-896ab7a1b6d4\") " pod="opendatahub/test-curl-7321-31dd25" Sep 16 13:43:54.457265 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:54.457183 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-31dd25" Sep 16 13:43:54.552434 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:54.552387 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:54.633025 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:54.632981 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7321-619a5e" podUID="74d35d0b-69ce-40f0-9fd6-c40f3efb0add" containerName="test-curl-7321-619a5e" containerID="cri-o://bb3a70798466958cec549dc684442e6ea32105c1e66bdc556ba5a138a41a797c" gracePeriod=2 Sep 16 13:43:54.768880 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:54.768849 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7321-31dd25"] Sep 16 13:43:54.771868 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:43:54.771841 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c243a10_9a39_400c_8df2_896ab7a1b6d4.slice/crio-e302fd89a05a55a7587f9459e13dc4ed00e510634b0a98cbb428d3c8c11aa4e8 WatchSource:0}: Error finding container e302fd89a05a55a7587f9459e13dc4ed00e510634b0a98cbb428d3c8c11aa4e8: Status 404 returned error can't find the container with id e302fd89a05a55a7587f9459e13dc4ed00e510634b0a98cbb428d3c8c11aa4e8 Sep 16 13:43:54.809094 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:54.809070 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:55.073186 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:55.073103 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:55.639375 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:55.639338 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7321-31dd25" event={"ID":"8c243a10-9a39-400c-8df2-896ab7a1b6d4","Type":"ContainerStarted","Data":"0d3e55d261981567038e1957a98ef08f10832d7f676f271306e897ad9b4fd15b"} Sep 16 13:43:55.639375 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:55.639376 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7321-31dd25" event={"ID":"8c243a10-9a39-400c-8df2-896ab7a1b6d4","Type":"ContainerStarted","Data":"e302fd89a05a55a7587f9459e13dc4ed00e510634b0a98cbb428d3c8c11aa4e8"} Sep 16 13:43:56.063807 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:56.063774 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:56.220959 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:56.220926 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:56.824399 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:56.824376 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-619a5e" Sep 16 13:43:56.909747 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:56.909710 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sttj4\" (UniqueName: \"kubernetes.io/projected/74d35d0b-69ce-40f0-9fd6-c40f3efb0add-kube-api-access-sttj4\") pod \"74d35d0b-69ce-40f0-9fd6-c40f3efb0add\" (UID: \"74d35d0b-69ce-40f0-9fd6-c40f3efb0add\") " Sep 16 13:43:56.912038 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:56.912015 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d35d0b-69ce-40f0-9fd6-c40f3efb0add-kube-api-access-sttj4" (OuterVolumeSpecName: "kube-api-access-sttj4") pod "74d35d0b-69ce-40f0-9fd6-c40f3efb0add" (UID: "74d35d0b-69ce-40f0-9fd6-c40f3efb0add"). InnerVolumeSpecName "kube-api-access-sttj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:43:57.010639 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:57.010587 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sttj4\" (UniqueName: \"kubernetes.io/projected/74d35d0b-69ce-40f0-9fd6-c40f3efb0add-kube-api-access-sttj4\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:43:57.163008 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:57.162910 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7321-31dd25" podStartSLOduration=3.162892033 podStartE2EDuration="3.162892033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:55.683365222 +0000 UTC m=+783.784177937" watchObservedRunningTime="2026-09-16 13:43:57.162892033 +0000 UTC m=+785.263704748" Sep 16 13:43:57.163935 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:57.163912 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7321-31dd25"] Sep 16 13:43:57.195509 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:57.195477 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7321-31dd25"] Sep 16 13:43:57.301461 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:57.301411 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:57.505419 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:57.505387 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:57.649396 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:57.649358 2580 generic.go:356] "Generic (PLEG): container finished" podID="74d35d0b-69ce-40f0-9fd6-c40f3efb0add" containerID="bb3a70798466958cec549dc684442e6ea32105c1e66bdc556ba5a138a41a797c" exitCode=137 Sep 16 13:43:57.649396 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:57.649400 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-619a5e" Sep 16 13:43:57.649690 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:57.649465 2580 scope.go:117] "RemoveContainer" containerID="bb3a70798466958cec549dc684442e6ea32105c1e66bdc556ba5a138a41a797c" Sep 16 13:43:57.649751 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:57.649685 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7321-31dd25" podUID="8c243a10-9a39-400c-8df2-896ab7a1b6d4" containerName="test-curl-7321-31dd25" containerID="cri-o://0d3e55d261981567038e1957a98ef08f10832d7f676f271306e897ad9b4fd15b" gracePeriod=2 Sep 16 13:43:57.657779 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:57.657763 2580 scope.go:117] "RemoveContainer" containerID="bb3a70798466958cec549dc684442e6ea32105c1e66bdc556ba5a138a41a797c" Sep 16 13:43:57.658061 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:43:57.658039 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb3a70798466958cec549dc684442e6ea32105c1e66bdc556ba5a138a41a797c\": container with ID starting with bb3a70798466958cec549dc684442e6ea32105c1e66bdc556ba5a138a41a797c not found: ID does not exist" containerID="bb3a70798466958cec549dc684442e6ea32105c1e66bdc556ba5a138a41a797c" Sep 16 13:43:57.658139 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:57.658068 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3a70798466958cec549dc684442e6ea32105c1e66bdc556ba5a138a41a797c"} err="failed to get container status \"bb3a70798466958cec549dc684442e6ea32105c1e66bdc556ba5a138a41a797c\": rpc error: code = NotFound desc = could not find container \"bb3a70798466958cec549dc684442e6ea32105c1e66bdc556ba5a138a41a797c\": container with ID starting with bb3a70798466958cec549dc684442e6ea32105c1e66bdc556ba5a138a41a797c not found: ID does not exist" Sep 16 13:43:57.663731 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:43:57.663699 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7321-31dd25\" is forbidden: User \"system:node:ip-10-0-132-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-9.ec2.internal' and this object" podUID="8c243a10-9a39-400c-8df2-896ab7a1b6d4" pod="opendatahub/test-curl-7321-31dd25" Sep 16 13:43:57.666984 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:43:57.666963 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7321-31dd25\" is forbidden: User \"system:node:ip-10-0-132-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-9.ec2.internal' and this object" podUID="8c243a10-9a39-400c-8df2-896ab7a1b6d4" pod="opendatahub/test-curl-7321-31dd25" Sep 16 13:43:57.669230 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:43:57.669207 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7321-619a5e\" is forbidden: User \"system:node:ip-10-0-132-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-9.ec2.internal' and this object" podUID="74d35d0b-69ce-40f0-9fd6-c40f3efb0add" pod="opendatahub/test-curl-7321-619a5e" Sep 16 13:43:57.920624 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:57.920495 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:58.221250 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.221216 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:58.356811 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.356776 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74d35d0b-69ce-40f0-9fd6-c40f3efb0add" path="/var/lib/kubelet/pods/74d35d0b-69ce-40f0-9fd6-c40f3efb0add/volumes" Sep 16 13:43:58.481713 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.481632 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t"] Sep 16 13:43:58.481958 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.481946 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c243a10-9a39-400c-8df2-896ab7a1b6d4" containerName="test-curl-7321-31dd25" Sep 16 13:43:58.481997 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.481960 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c243a10-9a39-400c-8df2-896ab7a1b6d4" containerName="test-curl-7321-31dd25" Sep 16 13:43:58.482035 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.482011 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c243a10-9a39-400c-8df2-896ab7a1b6d4" containerName="test-curl-7321-31dd25" Sep 16 13:43:58.486438 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.486420 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.490790 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.490757 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 16 13:43:58.490915 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.490850 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-14f2a6-openshift-default-dockercfg-nz6gj\"" Sep 16 13:43:58.492594 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.492569 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:58.521797 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.521772 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t"] Sep 16 13:43:58.624134 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.624106 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-credential-socket\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.624290 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.624149 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-workload-socket\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.624290 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.624170 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-data\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.624290 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.624194 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-token\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.624290 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.624250 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-workload-certs\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.624290 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.624277 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpbm8\" (UniqueName: \"kubernetes.io/projected/582d3a17-df42-4551-b213-9fc1a00a77d9-kube-api-access-xpbm8\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.624491 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.624303 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/582d3a17-df42-4551-b213-9fc1a00a77d9-istiod-ca-cert\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.624491 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.624321 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-envoy\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.624491 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.624347 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-podinfo\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.703085 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.703034 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:58.725817 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.725783 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/582d3a17-df42-4551-b213-9fc1a00a77d9-istiod-ca-cert\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.725974 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.725820 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-envoy\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.725974 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.725845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-podinfo\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.725974 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.725882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-credential-socket\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.725974 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.725931 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-workload-socket\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.725974 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.725961 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-data\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.726587 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.725985 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-token\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.726587 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.726061 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-workload-certs\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.726587 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.726087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpbm8\" (UniqueName: \"kubernetes.io/projected/582d3a17-df42-4551-b213-9fc1a00a77d9-kube-api-access-xpbm8\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.726587 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.726278 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-credential-socket\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.726587 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.726363 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-data\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.726587 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.726423 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-workload-socket\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.726587 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.726443 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-workload-certs\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.726587 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.726536 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/582d3a17-df42-4551-b213-9fc1a00a77d9-istiod-ca-cert\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.728496 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.728474 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-envoy\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.728631 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.728598 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-podinfo\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.747792 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.747737 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-token\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.749876 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.749853 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpbm8\" (UniqueName: \"kubernetes.io/projected/582d3a17-df42-4551-b213-9fc1a00a77d9-kube-api-access-xpbm8\") pod \"e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.783775 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.783744 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc"] Sep 16 13:43:58.788912 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.788342 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc" Sep 16 13:43:58.796170 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.796157 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:43:58.797226 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.797203 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-46ccbd\"" Sep 16 13:43:58.797454 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.797438 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-46ccbd-dockercfg-6p59g\"" Sep 16 13:43:58.822647 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.822585 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc"] Sep 16 13:43:58.928335 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.928306 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2741e4f1-17b2-4882-b6f2-1bcbf877666c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc\" (UID: \"2741e4f1-17b2-4882-b6f2-1bcbf877666c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc" Sep 16 13:43:58.928510 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.928353 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h7zg\" (UniqueName: \"kubernetes.io/projected/2741e4f1-17b2-4882-b6f2-1bcbf877666c-kube-api-access-4h7zg\") pod \"payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc\" (UID: \"2741e4f1-17b2-4882-b6f2-1bcbf877666c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc" Sep 16 13:43:58.961205 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:58.961165 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:59.029111 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:59.029039 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2741e4f1-17b2-4882-b6f2-1bcbf877666c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc\" (UID: \"2741e4f1-17b2-4882-b6f2-1bcbf877666c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc" Sep 16 13:43:59.029111 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:59.029088 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h7zg\" (UniqueName: \"kubernetes.io/projected/2741e4f1-17b2-4882-b6f2-1bcbf877666c-kube-api-access-4h7zg\") pod \"payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc\" (UID: \"2741e4f1-17b2-4882-b6f2-1bcbf877666c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc" Sep 16 13:43:59.029795 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:59.029777 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2741e4f1-17b2-4882-b6f2-1bcbf877666c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc\" (UID: \"2741e4f1-17b2-4882-b6f2-1bcbf877666c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc" Sep 16 13:43:59.043889 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:59.043857 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h7zg\" (UniqueName: \"kubernetes.io/projected/2741e4f1-17b2-4882-b6f2-1bcbf877666c-kube-api-access-4h7zg\") pod \"payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc\" (UID: \"2741e4f1-17b2-4882-b6f2-1bcbf877666c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc" Sep 16 13:43:59.098807 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:59.098773 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc" Sep 16 13:43:59.110190 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:59.110104 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t"] Sep 16 13:43:59.112310 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:43:59.112285 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582d3a17_df42_4551_b213_9fc1a00a77d9.slice/crio-7783ced512ec1c95c3e44eba4dd912cf3a706a3d34f52a974973095d69f00a71 WatchSource:0}: Error finding container 7783ced512ec1c95c3e44eba4dd912cf3a706a3d34f52a974973095d69f00a71: Status 404 returned error can't find the container with id 7783ced512ec1c95c3e44eba4dd912cf3a706a3d34f52a974973095d69f00a71 Sep 16 13:43:59.397751 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:59.397724 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc"] Sep 16 13:43:59.398687 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:43:59.398658 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2741e4f1_17b2_4882_b6f2_1bcbf877666c.slice/crio-2ca9463eddf728edff863064b5c9e366486ee1bd7c57d58819a2224e781d1423 WatchSource:0}: Error finding container 2ca9463eddf728edff863064b5c9e366486ee1bd7c57d58819a2224e781d1423: Status 404 returned error can't find the container with id 2ca9463eddf728edff863064b5c9e366486ee1bd7c57d58819a2224e781d1423 Sep 16 13:43:59.451162 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:59.451135 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:59.660801 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:59.660766 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:59.665649 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:59.665600 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" event={"ID":"582d3a17-df42-4551-b213-9fc1a00a77d9","Type":"ContainerStarted","Data":"7783ced512ec1c95c3e44eba4dd912cf3a706a3d34f52a974973095d69f00a71"} Sep 16 13:43:59.666905 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:59.666881 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc" event={"ID":"2741e4f1-17b2-4882-b6f2-1bcbf877666c","Type":"ContainerStarted","Data":"2ca9463eddf728edff863064b5c9e366486ee1bd7c57d58819a2224e781d1423"} Sep 16 13:43:59.847843 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:59.847815 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-31dd25" Sep 16 13:43:59.938577 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:59.938492 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgltd\" (UniqueName: \"kubernetes.io/projected/8c243a10-9a39-400c-8df2-896ab7a1b6d4-kube-api-access-mgltd\") pod \"8c243a10-9a39-400c-8df2-896ab7a1b6d4\" (UID: \"8c243a10-9a39-400c-8df2-896ab7a1b6d4\") " Sep 16 13:43:59.940913 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:43:59.940881 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c243a10-9a39-400c-8df2-896ab7a1b6d4-kube-api-access-mgltd" (OuterVolumeSpecName: "kube-api-access-mgltd") pod "8c243a10-9a39-400c-8df2-896ab7a1b6d4" (UID: "8c243a10-9a39-400c-8df2-896ab7a1b6d4"). InnerVolumeSpecName "kube-api-access-mgltd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:44:00.039962 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:00.039925 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgltd\" (UniqueName: \"kubernetes.io/projected/8c243a10-9a39-400c-8df2-896ab7a1b6d4-kube-api-access-mgltd\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:44:00.225466 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:00.225430 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:00.356274 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:00.356236 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c243a10-9a39-400c-8df2-896ab7a1b6d4" path="/var/lib/kubelet/pods/8c243a10-9a39-400c-8df2-896ab7a1b6d4/volumes" Sep 16 13:44:00.674076 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:00.673980 2580 generic.go:356] "Generic (PLEG): container finished" podID="8c243a10-9a39-400c-8df2-896ab7a1b6d4" containerID="0d3e55d261981567038e1957a98ef08f10832d7f676f271306e897ad9b4fd15b" exitCode=137 Sep 16 13:44:00.674076 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:00.674031 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-31dd25" Sep 16 13:44:00.674557 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:00.674083 2580 scope.go:117] "RemoveContainer" containerID="0d3e55d261981567038e1957a98ef08f10832d7f676f271306e897ad9b4fd15b" Sep 16 13:44:01.059574 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:01.059541 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:01.338109 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:01.338025 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:01.712504 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:01.712282 2580 scope.go:117] "RemoveContainer" containerID="0d3e55d261981567038e1957a98ef08f10832d7f676f271306e897ad9b4fd15b" Sep 16 13:44:01.712789 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:44:01.712680 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d3e55d261981567038e1957a98ef08f10832d7f676f271306e897ad9b4fd15b\": container with ID starting with 0d3e55d261981567038e1957a98ef08f10832d7f676f271306e897ad9b4fd15b not found: ID does not exist" containerID="0d3e55d261981567038e1957a98ef08f10832d7f676f271306e897ad9b4fd15b" Sep 16 13:44:01.712789 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:01.712708 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d3e55d261981567038e1957a98ef08f10832d7f676f271306e897ad9b4fd15b"} err="failed to get container status \"0d3e55d261981567038e1957a98ef08f10832d7f676f271306e897ad9b4fd15b\": rpc error: code = NotFound desc = could not find container \"0d3e55d261981567038e1957a98ef08f10832d7f676f271306e897ad9b4fd15b\": container with ID starting with 0d3e55d261981567038e1957a98ef08f10832d7f676f271306e897ad9b4fd15b not found: ID does not exist" Sep 16 13:44:01.728859 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:01.728808 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:44:01.728944 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:01.728925 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:44:01.728983 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:01.728969 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:44:02.228837 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:02.228807 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:02.684459 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:02.684364 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" event={"ID":"582d3a17-df42-4551-b213-9fc1a00a77d9","Type":"ContainerStarted","Data":"76bdcd11c690c4f5af4657e4d823983edccc51dd4ede8ffd64f903673ea37e81"} Sep 16 13:44:02.686706 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:02.686681 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc" event={"ID":"2741e4f1-17b2-4882-b6f2-1bcbf877666c","Type":"ContainerStarted","Data":"75ebf02d014bb61af2d0c0a03d35e57e79fa1158d2c14e45499bd0ccac69027b"} Sep 16 13:44:02.686855 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:02.686775 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc" Sep 16 13:44:02.750227 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:02.750165 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" podStartSLOduration=2.136005152 podStartE2EDuration="4.750145908s" totalImagesPullingTime="2.614140756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:02.749380922 +0000 UTC m=+790.850193636" watchObservedRunningTime="2026-09-16 13:44:02.750145908 +0000 UTC m=+790.850958624" Sep 16 13:44:02.797205 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:02.797167 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:44:02.802512 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:02.802484 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:44:02.855077 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:02.855025 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc" podStartSLOduration=2.536568422 podStartE2EDuration="4.855009217s" totalImagesPullingTime="2.318440795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:02.799634661 +0000 UTC m=+790.900447377" watchObservedRunningTime="2026-09-16 13:44:02.855009217 +0000 UTC m=+790.955821932" Sep 16 13:44:03.691577 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:03.691546 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:44:03.692553 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:03.692529 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:44:03.852456 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:03.852424 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:04.350075 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:04.350047 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:04.643829 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:04.643753 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:04.794430 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:04.794396 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:05.145884 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.145847 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:05.156323 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.156295 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd"] Sep 16 13:44:05.159810 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.159792 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.163135 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.163106 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4e1907f7-openshift-default-dockercfg-bk4ft\"" Sep 16 13:44:05.206253 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.206223 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd"] Sep 16 13:44:05.288711 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.288677 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-credential-socket\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.288869 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.288722 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-data\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.288869 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.288743 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-podinfo\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.288869 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.288769 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94brq\" (UniqueName: \"kubernetes.io/projected/cd465c3e-fb68-4bf9-b231-5ee5335e368d-kube-api-access-94brq\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.288869 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.288838 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-workload-certs\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.289048 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.288874 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-envoy\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.289048 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.288898 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istiod-ca-cert\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.289048 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.288925 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-token\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.289048 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.288959 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-workload-socket\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.389405 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.389376 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94brq\" (UniqueName: \"kubernetes.io/projected/cd465c3e-fb68-4bf9-b231-5ee5335e368d-kube-api-access-94brq\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.389555 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.389411 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-workload-certs\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.389555 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.389430 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-envoy\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.389555 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.389455 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istiod-ca-cert\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.389555 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.389478 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-token\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.389555 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.389519 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-workload-socket\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.389852 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.389564 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-credential-socket\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.389852 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.389638 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-data\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.389852 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.389669 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-podinfo\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.390063 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.389992 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-workload-certs\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.390063 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.390011 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-workload-socket\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.390063 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.389996 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-credential-socket\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.390218 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.390075 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-data\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.390278 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.390246 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istiod-ca-cert\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.391806 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.391784 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-envoy\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.392178 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.392161 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-podinfo\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.418929 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.418866 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94brq\" (UniqueName: \"kubernetes.io/projected/cd465c3e-fb68-4bf9-b231-5ee5335e368d-kube-api-access-94brq\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.443322 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.443290 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-token\") pod \"e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.450178 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.450153 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:05.471881 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.471851 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:05.668069 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.668040 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:05.780419 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.780397 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd"] Sep 16 13:44:05.782973 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:44:05.782952 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd465c3e_fb68_4bf9_b231_5ee5335e368d.slice/crio-852308be52ed45140cbd5d0d5e2afa3c1de84526aa2814f212439527ed064b9d WatchSource:0}: Error finding container 852308be52ed45140cbd5d0d5e2afa3c1de84526aa2814f212439527ed064b9d: Status 404 returned error can't find the container with id 852308be52ed45140cbd5d0d5e2afa3c1de84526aa2814f212439527ed064b9d Sep 16 13:44:05.786151 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.786120 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:44:05.786229 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.786185 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:44:05.786229 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.786212 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:44:05.978523 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:05.978492 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:06.708062 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:06.708028 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" event={"ID":"cd465c3e-fb68-4bf9-b231-5ee5335e368d","Type":"ContainerStarted","Data":"5ad6cd87509eca7dfe8ba7a513b524759a437c42e9f5d31023450eace6bac2fe"} Sep 16 13:44:06.708446 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:06.708081 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" event={"ID":"cd465c3e-fb68-4bf9-b231-5ee5335e368d","Type":"ContainerStarted","Data":"852308be52ed45140cbd5d0d5e2afa3c1de84526aa2814f212439527ed064b9d"} Sep 16 13:44:07.367140 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:07.367111 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:07.472999 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:07.472961 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:07.477944 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:07.477919 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:07.512521 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:07.512469 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" podStartSLOduration=2.512453519 podStartE2EDuration="2.512453519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:06.737887513 +0000 UTC m=+794.838700229" watchObservedRunningTime="2026-09-16 13:44:07.512453519 +0000 UTC m=+795.613266325" Sep 16 13:44:07.713486 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:07.713399 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:07.714370 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:07.714346 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:08.280702 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:08.280669 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:08.470767 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:08.470732 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:08.517711 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:08.517677 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:44:08.791215 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:08.791167 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:09.148385 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:09.148291 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:09.570022 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:09.569990 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:09.813336 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:09.813285 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:10.057639 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:10.057563 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:10.207796 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:10.207764 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:10.817529 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:10.817494 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:11.087411 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:11.087313 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:11.471941 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:11.471900 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:11.780485 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:11.780413 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:11.964173 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:11.964134 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:12.024636 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.024586 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt"] Sep 16 13:44:12.028526 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.028500 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.048367 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.048339 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1e5db0-openshift-default-dockercfg-92dpb\"" Sep 16 13:44:12.079326 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.079299 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98"] Sep 16 13:44:12.083324 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.083300 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt"] Sep 16 13:44:12.083425 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.083389 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98" Sep 16 13:44:12.089976 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.089941 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-14f2a6\"" Sep 16 13:44:12.096402 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.096368 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-14f2a6-dockercfg-pbb97\"" Sep 16 13:44:12.109262 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.109235 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98"] Sep 16 13:44:12.152571 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.152536 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istiod-ca-cert\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.152757 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.152581 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-data\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.152757 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.152647 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntpjj\" (UniqueName: \"kubernetes.io/projected/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-kube-api-access-ntpjj\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.152757 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.152731 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-workload-certs\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.152922 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.152757 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-envoy\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.152922 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.152804 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-podinfo\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.152922 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.152841 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmckj\" (UniqueName: \"kubernetes.io/projected/a413c643-7e13-4e01-9e54-5f6aab83e565-kube-api-access-lmckj\") pod \"payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98\" (UID: \"a413c643-7e13-4e01-9e54-5f6aab83e565\") " pod="openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98" Sep 16 13:44:12.152922 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.152865 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-token\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.153139 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.152977 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a413c643-7e13-4e01-9e54-5f6aab83e565-plugins-config-volume\") pod \"payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98\" (UID: \"a413c643-7e13-4e01-9e54-5f6aab83e565\") " pod="openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98" Sep 16 13:44:12.153139 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.153051 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-credential-socket\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.153139 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.153094 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-workload-socket\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.198042 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.198015 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:12.254473 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.254442 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-workload-certs\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.254473 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.254475 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-envoy\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.254705 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.254503 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-podinfo\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.254705 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.254538 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmckj\" (UniqueName: \"kubernetes.io/projected/a413c643-7e13-4e01-9e54-5f6aab83e565-kube-api-access-lmckj\") pod \"payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98\" (UID: \"a413c643-7e13-4e01-9e54-5f6aab83e565\") " pod="openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98" Sep 16 13:44:12.254705 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.254562 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-token\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.254705 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.254598 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a413c643-7e13-4e01-9e54-5f6aab83e565-plugins-config-volume\") pod \"payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98\" (UID: \"a413c643-7e13-4e01-9e54-5f6aab83e565\") " pod="openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98" Sep 16 13:44:12.254705 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.254675 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-credential-socket\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.254705 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.254702 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-workload-socket\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.255014 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.254733 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istiod-ca-cert\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.255014 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.254769 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-data\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.255014 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.254803 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntpjj\" (UniqueName: \"kubernetes.io/projected/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-kube-api-access-ntpjj\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.255014 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.254917 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-workload-certs\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.255223 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.255117 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-workload-socket\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.255223 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.255186 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-credential-socket\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.255414 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.255393 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-data\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.255527 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.255505 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a413c643-7e13-4e01-9e54-5f6aab83e565-plugins-config-volume\") pod \"payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98\" (UID: \"a413c643-7e13-4e01-9e54-5f6aab83e565\") " pod="openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98" Sep 16 13:44:12.255850 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.255826 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istiod-ca-cert\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.257031 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.257013 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-envoy\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.257280 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.257260 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-podinfo\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.269170 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.269141 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntpjj\" (UniqueName: \"kubernetes.io/projected/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-kube-api-access-ntpjj\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.279935 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.279906 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmckj\" (UniqueName: \"kubernetes.io/projected/a413c643-7e13-4e01-9e54-5f6aab83e565-kube-api-access-lmckj\") pod \"payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98\" (UID: \"a413c643-7e13-4e01-9e54-5f6aab83e565\") " pod="openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98" Sep 16 13:44:12.281701 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.281681 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-token\") pod \"e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.342945 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.342859 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:12.379964 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.379913 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:12.394776 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.394586 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98" Sep 16 13:44:12.574036 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.574001 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:12.648111 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.648081 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt"] Sep 16 13:44:12.649874 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:44:12.649848 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcef5277_4be2_4ffa_8dd1_b38c60f162aa.slice/crio-d83721e34fdf247f4a47cabc62b16f2d209db51d66333f1ffafd84998f939e8b WatchSource:0}: Error finding container d83721e34fdf247f4a47cabc62b16f2d209db51d66333f1ffafd84998f939e8b: Status 404 returned error can't find the container with id d83721e34fdf247f4a47cabc62b16f2d209db51d66333f1ffafd84998f939e8b Sep 16 13:44:12.653081 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.653048 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:44:12.653173 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.653118 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:44:12.653173 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.653158 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:44:12.703966 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:44:12.703937 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda413c643_7e13_4e01_9e54_5f6aab83e565.slice/crio-1253b362c2c98495a430bd53abfd64e64d8b2f89af16b34c13262bb033c71559 WatchSource:0}: Error finding container 1253b362c2c98495a430bd53abfd64e64d8b2f89af16b34c13262bb033c71559: Status 404 returned error can't find the container with id 1253b362c2c98495a430bd53abfd64e64d8b2f89af16b34c13262bb033c71559 Sep 16 13:44:12.704549 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.704498 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98"] Sep 16 13:44:12.742033 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.741992 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98" event={"ID":"a413c643-7e13-4e01-9e54-5f6aab83e565","Type":"ContainerStarted","Data":"1253b362c2c98495a430bd53abfd64e64d8b2f89af16b34c13262bb033c71559"} Sep 16 13:44:12.743963 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.743935 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" event={"ID":"fcef5277-4be2-4ffa-8dd1-b38c60f162aa","Type":"ContainerStarted","Data":"4d150fe896548d1e0990e83f971ef96f7ee22a37d8f761ad01bc0f2d6d713454"} Sep 16 13:44:12.744066 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.743984 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" event={"ID":"fcef5277-4be2-4ffa-8dd1-b38c60f162aa","Type":"ContainerStarted","Data":"d83721e34fdf247f4a47cabc62b16f2d209db51d66333f1ffafd84998f939e8b"} Sep 16 13:44:12.797066 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.797002 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" podStartSLOduration=1.796980591 podStartE2EDuration="1.796980591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:12.796429317 +0000 UTC m=+800.897242034" watchObservedRunningTime="2026-09-16 13:44:12.796980591 +0000 UTC m=+800.897793304" Sep 16 13:44:12.835928 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:12.835893 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:13.254412 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:13.254381 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:13.343711 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:13.343673 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:13.348784 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:13.348755 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:13.693582 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:13.693497 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc" Sep 16 13:44:13.750831 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:13.750793 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98" event={"ID":"a413c643-7e13-4e01-9e54-5f6aab83e565","Type":"ContainerStarted","Data":"a8a89269f10f552ae36e4ef6d6e5ad73c6f20d4f43b5ae33024085be75b54095"} Sep 16 13:44:13.751131 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:13.751108 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:13.751187 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:13.751163 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98" Sep 16 13:44:13.752141 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:13.752122 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:13.954445 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:13.954352 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98" podStartSLOduration=1.954336676 podStartE2EDuration="1.954336676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:13.952278392 +0000 UTC m=+802.053091107" watchObservedRunningTime="2026-09-16 13:44:13.954336676 +0000 UTC m=+802.055149390" Sep 16 13:44:15.493353 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:15.493324 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:15.694139 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:15.694102 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:16.603950 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:16.603914 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:16.909420 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:16.909336 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:17.190941 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:17.190847 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:17.740191 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:17.740116 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:17.889330 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:17.889302 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:18.637103 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:18.637068 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:18.881929 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:18.881896 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:19.083454 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:19.083418 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:19.705245 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:19.705209 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:21.013746 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:21.013712 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:22.291121 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:22.291087 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:23.593355 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:23.593323 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:23.829848 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:23.829820 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:24.093659 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:24.093598 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:24.549865 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:24.549829 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:24.754461 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:24.754429 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:24.757092 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:24.757070 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98" Sep 16 13:44:24.975532 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:24.975500 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:25.169479 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:25.169439 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5"] Sep 16 13:44:25.177102 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:25.177068 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5" Sep 16 13:44:25.181349 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:25.181321 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1e5db0\"" Sep 16 13:44:25.181933 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:25.181910 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1e5db0-dockercfg-gpxql\"" Sep 16 13:44:25.187188 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:25.187153 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5"] Sep 16 13:44:25.197562 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:25.197531 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28b781ab-6240-48ef-bb22-243ba2ea1428-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5\" (UID: \"28b781ab-6240-48ef-bb22-243ba2ea1428\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5" Sep 16 13:44:25.197562 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:25.197563 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nqdc\" (UniqueName: \"kubernetes.io/projected/28b781ab-6240-48ef-bb22-243ba2ea1428-kube-api-access-2nqdc\") pod \"payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5\" (UID: \"28b781ab-6240-48ef-bb22-243ba2ea1428\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5" Sep 16 13:44:25.298366 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:25.298279 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28b781ab-6240-48ef-bb22-243ba2ea1428-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5\" (UID: \"28b781ab-6240-48ef-bb22-243ba2ea1428\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5" Sep 16 13:44:25.298366 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:25.298315 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nqdc\" (UniqueName: \"kubernetes.io/projected/28b781ab-6240-48ef-bb22-243ba2ea1428-kube-api-access-2nqdc\") pod \"payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5\" (UID: \"28b781ab-6240-48ef-bb22-243ba2ea1428\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5" Sep 16 13:44:25.299019 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:25.299000 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28b781ab-6240-48ef-bb22-243ba2ea1428-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5\" (UID: \"28b781ab-6240-48ef-bb22-243ba2ea1428\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5" Sep 16 13:44:25.304268 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:25.304245 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:25.307377 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:25.307354 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nqdc\" (UniqueName: \"kubernetes.io/projected/28b781ab-6240-48ef-bb22-243ba2ea1428-kube-api-access-2nqdc\") pod \"payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5\" (UID: \"28b781ab-6240-48ef-bb22-243ba2ea1428\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5" Sep 16 13:44:25.495755 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:25.495712 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5" Sep 16 13:44:25.787585 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:25.787557 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5"] Sep 16 13:44:25.790760 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:44:25.790729 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b781ab_6240_48ef_bb22_243ba2ea1428.slice/crio-7c0194c3e616727beebcd87be233465bce127ea190a3861e24daf92b395fa411 WatchSource:0}: Error finding container 7c0194c3e616727beebcd87be233465bce127ea190a3861e24daf92b395fa411: Status 404 returned error can't find the container with id 7c0194c3e616727beebcd87be233465bce127ea190a3861e24daf92b395fa411 Sep 16 13:44:25.808171 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:25.808107 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5" event={"ID":"28b781ab-6240-48ef-bb22-243ba2ea1428","Type":"ContainerStarted","Data":"7c0194c3e616727beebcd87be233465bce127ea190a3861e24daf92b395fa411"} Sep 16 13:44:26.205042 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:26.204960 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:26.439885 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:26.439849 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:26.586393 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:26.586299 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:26.762048 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:26.762014 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:26.814312 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:26.814279 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5" event={"ID":"28b781ab-6240-48ef-bb22-243ba2ea1428","Type":"ContainerStarted","Data":"cb6d61ced61f34ea96860bef338fd47aae2871867686e9c1da1a185a32522f8f"} Sep 16 13:44:26.814770 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:26.814440 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5" Sep 16 13:44:26.839722 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:26.839587 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5" podStartSLOduration=1.8395661589999999 podStartE2EDuration="1.839566159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:26.838172681 +0000 UTC m=+814.938985396" watchObservedRunningTime="2026-09-16 13:44:26.839566159 +0000 UTC m=+814.940378875" Sep 16 13:44:27.799805 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:27.799767 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:28.557323 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:28.557276 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:44:29.226659 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:29.226634 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:29.829830 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:29.829791 2580 generic.go:356] "Generic (PLEG): container finished" podID="582d3a17-df42-4551-b213-9fc1a00a77d9" containerID="76bdcd11c690c4f5af4657e4d823983edccc51dd4ede8ffd64f903673ea37e81" exitCode=0 Sep 16 13:44:29.829830 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:29.829832 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" event={"ID":"582d3a17-df42-4551-b213-9fc1a00a77d9","Type":"ContainerDied","Data":"76bdcd11c690c4f5af4657e4d823983edccc51dd4ede8ffd64f903673ea37e81"} Sep 16 13:44:29.830268 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:29.830187 2580 scope.go:117] "RemoveContainer" containerID="76bdcd11c690c4f5af4657e4d823983edccc51dd4ede8ffd64f903673ea37e81" Sep 16 13:44:29.831345 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:29.831316 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:44:29.831403 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:29.831387 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:44:29.831438 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:29.831418 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:44:30.576519 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:30.576481 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:30.836762 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:30.836679 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" event={"ID":"582d3a17-df42-4551-b213-9fc1a00a77d9","Type":"ContainerStarted","Data":"1bcdaf348d5c9578ad70f52878963ce1c3eb9c545cf55fcd7ce89b2da3641351"} Sep 16 13:44:31.405967 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:31.405932 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:31.799638 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:31.799028 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:44:31.799638 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:31.799166 2580 patch_prober.go:28] interesting pod/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 16 13:44:31.799638 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:31.799220 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" podUID="582d3a17-df42-4551-b213-9fc1a00a77d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 16 13:44:32.187993 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:32.187889 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:32.797340 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:32.797294 2580 patch_prober.go:28] interesting pod/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 16 13:44:32.797693 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:32.797635 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" podUID="582d3a17-df42-4551-b213-9fc1a00a77d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 16 13:44:33.319029 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:33.318991 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:33.692722 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:33.692593 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:44:33.785455 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:33.785412 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e1907f7-cnt86"] Sep 16 13:44:33.788621 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:33.788574 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e1907f7-cnt86" Sep 16 13:44:33.794313 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:33.794275 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:33.794547 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:33.794526 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:44:33.794875 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:33.794759 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-thk5m\"" Sep 16 13:44:33.796788 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:33.796762 2580 patch_prober.go:28] interesting pod/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 16 13:44:33.796893 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:33.796830 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" podUID="582d3a17-df42-4551-b213-9fc1a00a77d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 16 13:44:33.807011 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:33.806981 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e1907f7-cnt86"] Sep 16 13:44:33.896633 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:33.896407 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59e4359-81cf-49d5-92e3-0fe165bebc9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e1907f7-cnt86\" (UID: \"f59e4359-81cf-49d5-92e3-0fe165bebc9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e1907f7-cnt86" Sep 16 13:44:33.903834 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:33.903796 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:33.997543 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:33.997490 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59e4359-81cf-49d5-92e3-0fe165bebc9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e1907f7-cnt86\" (UID: \"f59e4359-81cf-49d5-92e3-0fe165bebc9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e1907f7-cnt86" Sep 16 13:44:33.998433 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:33.998397 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59e4359-81cf-49d5-92e3-0fe165bebc9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e1907f7-cnt86\" (UID: \"f59e4359-81cf-49d5-92e3-0fe165bebc9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e1907f7-cnt86" Sep 16 13:44:34.108365 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:34.108321 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e1907f7-cnt86" Sep 16 13:44:34.442633 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:34.442168 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:34.498628 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:34.497991 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e1907f7-cnt86"] Sep 16 13:44:34.797517 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:34.797407 2580 patch_prober.go:28] interesting pod/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 16 13:44:34.797517 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:34.797482 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" podUID="582d3a17-df42-4551-b213-9fc1a00a77d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 16 13:44:34.859430 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:34.859395 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e1907f7-cnt86" event={"ID":"f59e4359-81cf-49d5-92e3-0fe165bebc9a","Type":"ContainerStarted","Data":"b6b4adb2a71411d6b5396d7a2dc51e578b1e6807cef4efcc2c8892d4926f2c5c"} Sep 16 13:44:35.256062 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:35.256028 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:35.369786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:35.369741 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:35.802197 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:35.801725 2580 patch_prober.go:28] interesting pod/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 16 13:44:35.802197 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:35.801793 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" podUID="582d3a17-df42-4551-b213-9fc1a00a77d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 16 13:44:36.148368 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:36.148335 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:36.802369 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:36.802325 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:44:36.803691 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:36.803667 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:44:36.944329 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:36.944293 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:37.821114 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:37.821079 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5" Sep 16 13:44:37.843858 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:37.843820 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:37.889657 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:37.889585 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e1907f7-cnt86" event={"ID":"f59e4359-81cf-49d5-92e3-0fe165bebc9a","Type":"ContainerStarted","Data":"9ed29868e6f2c697cbec7efa6ebe68f146c09f118981c545159e6764c53e6fa4"} Sep 16 13:44:37.912455 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:37.912391 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e1907f7-cnt86" podStartSLOduration=2.000533479 podStartE2EDuration="4.912368311s" totalImagesPullingTime="2.911834832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:37.910947904 +0000 UTC m=+826.011760634" watchObservedRunningTime="2026-09-16 13:44:37.912368311 +0000 UTC m=+826.013181030" Sep 16 13:44:38.600965 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:38.600870 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:38.754777 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:38.754734 2580 patch_prober.go:28] interesting pod/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 16 13:44:38.754952 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:38.754808 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" podUID="cd465c3e-fb68-4bf9-b231-5ee5335e368d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 16 13:44:38.896631 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:38.896510 2580 generic.go:356] "Generic (PLEG): container finished" podID="f59e4359-81cf-49d5-92e3-0fe165bebc9a" containerID="9ed29868e6f2c697cbec7efa6ebe68f146c09f118981c545159e6764c53e6fa4" exitCode=0 Sep 16 13:44:38.897009 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:38.896631 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e1907f7-cnt86" event={"ID":"f59e4359-81cf-49d5-92e3-0fe165bebc9a","Type":"ContainerDied","Data":"9ed29868e6f2c697cbec7efa6ebe68f146c09f118981c545159e6764c53e6fa4"} Sep 16 13:44:39.333082 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:39.333045 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:39.902619 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:39.902574 2580 generic.go:356] "Generic (PLEG): container finished" podID="cd465c3e-fb68-4bf9-b231-5ee5335e368d" containerID="5ad6cd87509eca7dfe8ba7a513b524759a437c42e9f5d31023450eace6bac2fe" exitCode=0 Sep 16 13:44:39.903102 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:39.902651 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" event={"ID":"cd465c3e-fb68-4bf9-b231-5ee5335e368d","Type":"ContainerDied","Data":"5ad6cd87509eca7dfe8ba7a513b524759a437c42e9f5d31023450eace6bac2fe"} Sep 16 13:44:39.903236 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:39.903218 2580 scope.go:117] "RemoveContainer" containerID="5ad6cd87509eca7dfe8ba7a513b524759a437c42e9f5d31023450eace6bac2fe" Sep 16 13:44:39.904345 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:39.904321 2580 generic.go:356] "Generic (PLEG): container finished" podID="fcef5277-4be2-4ffa-8dd1-b38c60f162aa" containerID="4d150fe896548d1e0990e83f971ef96f7ee22a37d8f761ad01bc0f2d6d713454" exitCode=0 Sep 16 13:44:39.904474 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:39.904420 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" event={"ID":"fcef5277-4be2-4ffa-8dd1-b38c60f162aa","Type":"ContainerDied","Data":"4d150fe896548d1e0990e83f971ef96f7ee22a37d8f761ad01bc0f2d6d713454"} Sep 16 13:44:39.904545 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:39.904500 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:44:39.904646 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:39.904595 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:44:39.904777 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:39.904672 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:44:39.904869 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:39.904853 2580 scope.go:117] "RemoveContainer" containerID="4d150fe896548d1e0990e83f971ef96f7ee22a37d8f761ad01bc0f2d6d713454" Sep 16 13:44:39.906048 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:39.906019 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:44:39.906133 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:39.906116 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:44:39.906189 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:39.906175 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:44:40.054804 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:40.054498 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e1907f7-cnt86" Sep 16 13:44:40.160081 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:40.159995 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59e4359-81cf-49d5-92e3-0fe165bebc9a-maas-api-service-ca\") pod \"f59e4359-81cf-49d5-92e3-0fe165bebc9a\" (UID: \"f59e4359-81cf-49d5-92e3-0fe165bebc9a\") " Sep 16 13:44:40.160436 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:40.160402 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f59e4359-81cf-49d5-92e3-0fe165bebc9a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f59e4359-81cf-49d5-92e3-0fe165bebc9a" (UID: "f59e4359-81cf-49d5-92e3-0fe165bebc9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:44:40.261338 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:40.261300 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59e4359-81cf-49d5-92e3-0fe165bebc9a-maas-api-service-ca\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:44:40.910047 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:40.910010 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e1907f7-cnt86" Sep 16 13:44:40.910047 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:40.910036 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e1907f7-cnt86" event={"ID":"f59e4359-81cf-49d5-92e3-0fe165bebc9a","Type":"ContainerDied","Data":"b6b4adb2a71411d6b5396d7a2dc51e578b1e6807cef4efcc2c8892d4926f2c5c"} Sep 16 13:44:40.910498 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:40.910090 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6b4adb2a71411d6b5396d7a2dc51e578b1e6807cef4efcc2c8892d4926f2c5c" Sep 16 13:44:40.912282 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:40.912253 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" event={"ID":"cd465c3e-fb68-4bf9-b231-5ee5335e368d","Type":"ContainerStarted","Data":"7991c8623170b9b8d3ba96a59652a75e2ae11539cba6278fdd034519f2675c44"} Sep 16 13:44:40.914121 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:40.914099 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" event={"ID":"fcef5277-4be2-4ffa-8dd1-b38c60f162aa","Type":"ContainerStarted","Data":"a6b33fe23eec663ce3ce4b8f36815991855cab9920d7b83c1f97a9cf5b1bc6ae"} Sep 16 13:44:41.343318 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:41.343279 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:41.344442 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:41.344416 2580 patch_prober.go:28] interesting pod/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Sep 16 13:44:41.344543 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:41.344472 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" podUID="fcef5277-4be2-4ffa-8dd1-b38c60f162aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Sep 16 13:44:41.472761 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:41.472725 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:41.473080 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:41.473046 2580 patch_prober.go:28] interesting pod/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 16 13:44:41.473173 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:41.473130 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" podUID="cd465c3e-fb68-4bf9-b231-5ee5335e368d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 16 13:44:42.343400 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:42.343356 2580 patch_prober.go:28] interesting pod/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Sep 16 13:44:42.343847 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:42.343428 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" podUID="fcef5277-4be2-4ffa-8dd1-b38c60f162aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Sep 16 13:44:42.473335 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:42.473292 2580 patch_prober.go:28] interesting pod/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 16 13:44:42.473544 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:42.473372 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" podUID="cd465c3e-fb68-4bf9-b231-5ee5335e368d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 16 13:44:43.343627 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:43.343566 2580 patch_prober.go:28] interesting pod/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Sep 16 13:44:43.343985 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:43.343664 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" podUID="fcef5277-4be2-4ffa-8dd1-b38c60f162aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Sep 16 13:44:43.473110 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:43.473062 2580 patch_prober.go:28] interesting pod/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 16 13:44:43.473304 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:43.473170 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" podUID="cd465c3e-fb68-4bf9-b231-5ee5335e368d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 16 13:44:43.751766 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:43.751730 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:44.344018 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:44.343987 2580 patch_prober.go:28] interesting pod/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Sep 16 13:44:44.344477 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:44.344449 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" podUID="fcef5277-4be2-4ffa-8dd1-b38c60f162aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Sep 16 13:44:44.472776 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:44.472736 2580 patch_prober.go:28] interesting pod/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 16 13:44:44.473063 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:44.473034 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" podUID="cd465c3e-fb68-4bf9-b231-5ee5335e368d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 16 13:44:45.230322 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:45.230293 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:45.343550 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:45.343507 2580 patch_prober.go:28] interesting pod/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Sep 16 13:44:45.343752 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:45.343560 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" podUID="fcef5277-4be2-4ffa-8dd1-b38c60f162aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Sep 16 13:44:45.473384 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:45.473335 2580 patch_prober.go:28] interesting pod/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 16 13:44:45.473907 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:45.473407 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" podUID="cd465c3e-fb68-4bf9-b231-5ee5335e368d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 16 13:44:46.344212 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:46.344178 2580 patch_prober.go:28] interesting pod/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Sep 16 13:44:46.344385 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:46.344235 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" podUID="fcef5277-4be2-4ffa-8dd1-b38c60f162aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Sep 16 13:44:46.481887 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:46.481841 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:46.483129 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:46.483102 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:46.486046 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:46.486014 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:47.347348 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:47.347315 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:47.348416 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:47.348393 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:44:50.351983 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:50.351945 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:51.039738 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:51.039710 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:51.175070 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:51.175043 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:51.284170 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:51.284141 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:51.342683 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:51.342574 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e1907f7-cnt86"] Sep 16 13:44:51.346127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:51.346096 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e1907f7-cnt86"] Sep 16 13:44:51.376327 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:51.376282 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:51.488500 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:51.488471 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd"] Sep 16 13:44:51.488747 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:51.488724 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" podUID="cd465c3e-fb68-4bf9-b231-5ee5335e368d" containerName="istio-proxy" containerID="cri-o://7991c8623170b9b8d3ba96a59652a75e2ae11539cba6278fdd034519f2675c44" gracePeriod=30 Sep 16 13:44:51.509175 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:51.509143 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:51.835764 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:51.835713 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:52.059321 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:52.059289 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:52.262160 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:52.262127 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:52.355947 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:52.355922 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f59e4359-81cf-49d5-92e3-0fe165bebc9a" path="/var/lib/kubelet/pods/f59e4359-81cf-49d5-92e3-0fe165bebc9a/volumes" Sep 16 13:44:52.386155 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:52.386124 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:52.731641 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:52.731582 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:52.888371 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:52.888340 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:53.123521 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:53.123427 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:54.029359 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:54.029327 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:54.905949 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:54.905915 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:55.699242 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:55.699207 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:55.799329 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:55.799298 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:55.928798 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:55.928765 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:56.739048 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.739026 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:56.820423 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.820329 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94brq\" (UniqueName: \"kubernetes.io/projected/cd465c3e-fb68-4bf9-b231-5ee5335e368d-kube-api-access-94brq\") pod \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " Sep 16 13:44:56.820423 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.820368 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-workload-certs\") pod \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " Sep 16 13:44:56.820702 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.820432 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-data\") pod \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " Sep 16 13:44:56.820702 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.820454 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-envoy\") pod \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " Sep 16 13:44:56.820702 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.820470 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-token\") pod \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " Sep 16 13:44:56.820702 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.820501 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istiod-ca-cert\") pod \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " Sep 16 13:44:56.820702 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.820534 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-podinfo\") pod \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " Sep 16 13:44:56.820702 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.820575 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-credential-socket\") pod \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " Sep 16 13:44:56.820702 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.820645 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-workload-socket\") pod \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\" (UID: \"cd465c3e-fb68-4bf9-b231-5ee5335e368d\") " Sep 16 13:44:56.821056 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.820797 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cd465c3e-fb68-4bf9-b231-5ee5335e368d" (UID: "cd465c3e-fb68-4bf9-b231-5ee5335e368d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:44:56.821056 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.820941 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cd465c3e-fb68-4bf9-b231-5ee5335e368d" (UID: "cd465c3e-fb68-4bf9-b231-5ee5335e368d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:44:56.821056 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.820827 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-data" (OuterVolumeSpecName: "istio-data") pod "cd465c3e-fb68-4bf9-b231-5ee5335e368d" (UID: "cd465c3e-fb68-4bf9-b231-5ee5335e368d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:44:56.821056 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.820970 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-workload-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:44:56.821056 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.821030 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cd465c3e-fb68-4bf9-b231-5ee5335e368d" (UID: "cd465c3e-fb68-4bf9-b231-5ee5335e368d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:44:56.821498 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.821098 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cd465c3e-fb68-4bf9-b231-5ee5335e368d" (UID: "cd465c3e-fb68-4bf9-b231-5ee5335e368d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:44:56.822973 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.822949 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cd465c3e-fb68-4bf9-b231-5ee5335e368d" (UID: "cd465c3e-fb68-4bf9-b231-5ee5335e368d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:44:56.823092 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.822977 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-token" (OuterVolumeSpecName: "istio-token") pod "cd465c3e-fb68-4bf9-b231-5ee5335e368d" (UID: "cd465c3e-fb68-4bf9-b231-5ee5335e368d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:44:56.823092 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.823043 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cd465c3e-fb68-4bf9-b231-5ee5335e368d" (UID: "cd465c3e-fb68-4bf9-b231-5ee5335e368d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 13:44:56.823092 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.823054 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd465c3e-fb68-4bf9-b231-5ee5335e368d-kube-api-access-94brq" (OuterVolumeSpecName: "kube-api-access-94brq") pod "cd465c3e-fb68-4bf9-b231-5ee5335e368d" (UID: "cd465c3e-fb68-4bf9-b231-5ee5335e368d"). InnerVolumeSpecName "kube-api-access-94brq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:44:56.922345 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.922310 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-data\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:44:56.922345 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.922339 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-envoy\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:44:56.922345 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.922350 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-token\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:44:56.922586 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.922358 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istiod-ca-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:44:56.922586 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.922367 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd465c3e-fb68-4bf9-b231-5ee5335e368d-istio-podinfo\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:44:56.922586 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.922374 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-credential-socket\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:44:56.922586 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.922383 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd465c3e-fb68-4bf9-b231-5ee5335e368d-workload-socket\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:44:56.922586 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.922391 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94brq\" (UniqueName: \"kubernetes.io/projected/cd465c3e-fb68-4bf9-b231-5ee5335e368d-kube-api-access-94brq\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:44:56.992434 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.992393 2580 generic.go:356] "Generic (PLEG): container finished" podID="cd465c3e-fb68-4bf9-b231-5ee5335e368d" containerID="7991c8623170b9b8d3ba96a59652a75e2ae11539cba6278fdd034519f2675c44" exitCode=0 Sep 16 13:44:56.992655 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.992443 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" Sep 16 13:44:56.992655 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.992481 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" event={"ID":"cd465c3e-fb68-4bf9-b231-5ee5335e368d","Type":"ContainerDied","Data":"7991c8623170b9b8d3ba96a59652a75e2ae11539cba6278fdd034519f2675c44"} Sep 16 13:44:56.992655 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.992541 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd" event={"ID":"cd465c3e-fb68-4bf9-b231-5ee5335e368d","Type":"ContainerDied","Data":"852308be52ed45140cbd5d0d5e2afa3c1de84526aa2814f212439527ed064b9d"} Sep 16 13:44:56.992655 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:56.992563 2580 scope.go:117] "RemoveContainer" containerID="7991c8623170b9b8d3ba96a59652a75e2ae11539cba6278fdd034519f2675c44" Sep 16 13:44:57.002091 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:57.002073 2580 scope.go:117] "RemoveContainer" containerID="5ad6cd87509eca7dfe8ba7a513b524759a437c42e9f5d31023450eace6bac2fe" Sep 16 13:44:57.010072 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:57.010056 2580 scope.go:117] "RemoveContainer" containerID="7991c8623170b9b8d3ba96a59652a75e2ae11539cba6278fdd034519f2675c44" Sep 16 13:44:57.010338 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:44:57.010321 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7991c8623170b9b8d3ba96a59652a75e2ae11539cba6278fdd034519f2675c44\": container with ID starting with 7991c8623170b9b8d3ba96a59652a75e2ae11539cba6278fdd034519f2675c44 not found: ID does not exist" containerID="7991c8623170b9b8d3ba96a59652a75e2ae11539cba6278fdd034519f2675c44" Sep 16 13:44:57.010383 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:57.010347 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7991c8623170b9b8d3ba96a59652a75e2ae11539cba6278fdd034519f2675c44"} err="failed to get container status \"7991c8623170b9b8d3ba96a59652a75e2ae11539cba6278fdd034519f2675c44\": rpc error: code = NotFound desc = could not find container \"7991c8623170b9b8d3ba96a59652a75e2ae11539cba6278fdd034519f2675c44\": container with ID starting with 7991c8623170b9b8d3ba96a59652a75e2ae11539cba6278fdd034519f2675c44 not found: ID does not exist" Sep 16 13:44:57.010383 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:57.010365 2580 scope.go:117] "RemoveContainer" containerID="5ad6cd87509eca7dfe8ba7a513b524759a437c42e9f5d31023450eace6bac2fe" Sep 16 13:44:57.010637 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:44:57.010619 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad6cd87509eca7dfe8ba7a513b524759a437c42e9f5d31023450eace6bac2fe\": container with ID starting with 5ad6cd87509eca7dfe8ba7a513b524759a437c42e9f5d31023450eace6bac2fe not found: ID does not exist" containerID="5ad6cd87509eca7dfe8ba7a513b524759a437c42e9f5d31023450eace6bac2fe" Sep 16 13:44:57.010703 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:57.010642 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad6cd87509eca7dfe8ba7a513b524759a437c42e9f5d31023450eace6bac2fe"} err="failed to get container status \"5ad6cd87509eca7dfe8ba7a513b524759a437c42e9f5d31023450eace6bac2fe\": rpc error: code = NotFound desc = could not find container \"5ad6cd87509eca7dfe8ba7a513b524759a437c42e9f5d31023450eace6bac2fe\": container with ID starting with 5ad6cd87509eca7dfe8ba7a513b524759a437c42e9f5d31023450eace6bac2fe not found: ID does not exist" Sep 16 13:44:57.018808 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:57.018748 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd"] Sep 16 13:44:57.020435 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:57.020413 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4e1907f7-openshift-default-6467649b6b-gwxkd"] Sep 16 13:44:57.022950 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:57.022927 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:57.401109 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:57.401074 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:44:57.744718 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:57.744684 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:57.845126 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:57.845099 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:58.089277 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.089193 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:58.331278 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.331241 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:58.355118 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.355043 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd465c3e-fb68-4bf9-b231-5ee5335e368d" path="/var/lib/kubelet/pods/cd465c3e-fb68-4bf9-b231-5ee5335e368d/volumes" Sep 16 13:44:58.373836 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.373812 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v"] Sep 16 13:44:58.374192 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.374179 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd465c3e-fb68-4bf9-b231-5ee5335e368d" containerName="istio-proxy" Sep 16 13:44:58.374240 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.374194 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd465c3e-fb68-4bf9-b231-5ee5335e368d" containerName="istio-proxy" Sep 16 13:44:58.374240 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.374209 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f59e4359-81cf-49d5-92e3-0fe165bebc9a" containerName="revoke-keys" Sep 16 13:44:58.374240 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.374214 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f59e4359-81cf-49d5-92e3-0fe165bebc9a" containerName="revoke-keys" Sep 16 13:44:58.374328 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.374280 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd465c3e-fb68-4bf9-b231-5ee5335e368d" containerName="istio-proxy" Sep 16 13:44:58.374328 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.374290 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f59e4359-81cf-49d5-92e3-0fe165bebc9a" containerName="revoke-keys" Sep 16 13:44:58.377037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.377021 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v" Sep 16 13:44:58.379701 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.379682 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-6d94e3f8\"" Sep 16 13:44:58.388389 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.388364 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v"] Sep 16 13:44:58.398055 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.398028 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t"] Sep 16 13:44:58.398410 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.398394 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd465c3e-fb68-4bf9-b231-5ee5335e368d" containerName="istio-proxy" Sep 16 13:44:58.398410 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.398410 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd465c3e-fb68-4bf9-b231-5ee5335e368d" containerName="istio-proxy" Sep 16 13:44:58.398564 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.398484 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd465c3e-fb68-4bf9-b231-5ee5335e368d" containerName="istio-proxy" Sep 16 13:44:58.403843 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.403823 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t" Sep 16 13:44:58.418646 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.407834 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-6d94e3f8-dockercfg-n2h45\"" Sep 16 13:44:58.418646 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.418086 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t"] Sep 16 13:44:58.427517 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.427495 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:58.536956 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.536918 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9wpx\" (UniqueName: \"kubernetes.io/projected/cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3-kube-api-access-s9wpx\") pod \"payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t\" (UID: \"cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3\") " pod="openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t" Sep 16 13:44:58.537140 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.536979 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09a01f3-292d-4460-b119-1906e3d45586-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v\" (UID: \"e09a01f3-292d-4460-b119-1906e3d45586\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v" Sep 16 13:44:58.537140 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.537009 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t\" (UID: \"cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3\") " pod="openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t" Sep 16 13:44:58.537140 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.537036 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxdc7\" (UniqueName: \"kubernetes.io/projected/e09a01f3-292d-4460-b119-1906e3d45586-kube-api-access-vxdc7\") pod \"payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v\" (UID: \"e09a01f3-292d-4460-b119-1906e3d45586\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v" Sep 16 13:44:58.637903 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.637813 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t\" (UID: \"cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3\") " pod="openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t" Sep 16 13:44:58.637903 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.637855 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxdc7\" (UniqueName: \"kubernetes.io/projected/e09a01f3-292d-4460-b119-1906e3d45586-kube-api-access-vxdc7\") pod \"payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v\" (UID: \"e09a01f3-292d-4460-b119-1906e3d45586\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v" Sep 16 13:44:58.638127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.637929 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9wpx\" (UniqueName: \"kubernetes.io/projected/cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3-kube-api-access-s9wpx\") pod \"payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t\" (UID: \"cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3\") " pod="openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t" Sep 16 13:44:58.638127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.637964 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09a01f3-292d-4460-b119-1906e3d45586-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v\" (UID: \"e09a01f3-292d-4460-b119-1906e3d45586\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v" Sep 16 13:44:58.638567 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.638544 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t\" (UID: \"cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3\") " pod="openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t" Sep 16 13:44:58.638662 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.638578 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09a01f3-292d-4460-b119-1906e3d45586-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v\" (UID: \"e09a01f3-292d-4460-b119-1906e3d45586\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v" Sep 16 13:44:58.647343 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.647315 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxdc7\" (UniqueName: \"kubernetes.io/projected/e09a01f3-292d-4460-b119-1906e3d45586-kube-api-access-vxdc7\") pod \"payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v\" (UID: \"e09a01f3-292d-4460-b119-1906e3d45586\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v" Sep 16 13:44:58.647449 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.647324 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9wpx\" (UniqueName: \"kubernetes.io/projected/cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3-kube-api-access-s9wpx\") pod \"payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t\" (UID: \"cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3\") " pod="openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t" Sep 16 13:44:58.687394 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.687370 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v" Sep 16 13:44:58.727835 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.727793 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t" Sep 16 13:44:58.818660 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.818615 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:58.961696 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:58.961668 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v"] Sep 16 13:44:58.963834 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:44:58.963806 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode09a01f3_292d_4460_b119_1906e3d45586.slice/crio-20e816e2b467184021172994d36b28510e66c495aa9d46e48eb1592761a6e42c WatchSource:0}: Error finding container 20e816e2b467184021172994d36b28510e66c495aa9d46e48eb1592761a6e42c: Status 404 returned error can't find the container with id 20e816e2b467184021172994d36b28510e66c495aa9d46e48eb1592761a6e42c Sep 16 13:44:59.003484 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:59.003454 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v" event={"ID":"e09a01f3-292d-4460-b119-1906e3d45586","Type":"ContainerStarted","Data":"20e816e2b467184021172994d36b28510e66c495aa9d46e48eb1592761a6e42c"} Sep 16 13:44:59.011526 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:59.011500 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t"] Sep 16 13:44:59.014567 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:44:59.014545 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc69ec3d_b78c_4073_ae1a_c9bd2a13fca3.slice/crio-6fec89e09f1eaf6b890ddc7dcf6007c57020d2778e4292df337cbc9095096ab4 WatchSource:0}: Error finding container 6fec89e09f1eaf6b890ddc7dcf6007c57020d2778e4292df337cbc9095096ab4: Status 404 returned error can't find the container with id 6fec89e09f1eaf6b890ddc7dcf6007c57020d2778e4292df337cbc9095096ab4 Sep 16 13:44:59.688482 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:44:59.688450 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:00.010849 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.010814 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t" event={"ID":"cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3","Type":"ContainerStarted","Data":"af7b86e279f6f8c63d57139bc264921917c2d3ad83dd240c80a246caea83daee"} Sep 16 13:45:00.011308 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.010880 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t" Sep 16 13:45:00.011308 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.010905 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t" event={"ID":"cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3","Type":"ContainerStarted","Data":"6fec89e09f1eaf6b890ddc7dcf6007c57020d2778e4292df337cbc9095096ab4"} Sep 16 13:45:00.012487 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.012467 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v" event={"ID":"e09a01f3-292d-4460-b119-1906e3d45586","Type":"ContainerStarted","Data":"7d999a3051a7da1f6ca0df7f3e0087aa61c66f0b9788a373314442583002e448"} Sep 16 13:45:00.012628 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.012594 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v" Sep 16 13:45:00.033260 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.033210 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t" podStartSLOduration=2.033193244 podStartE2EDuration="2.033193244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:00.031302218 +0000 UTC m=+848.132114936" watchObservedRunningTime="2026-09-16 13:45:00.033193244 +0000 UTC m=+848.134005958" Sep 16 13:45:00.180725 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.180666 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v" podStartSLOduration=2.180642046 podStartE2EDuration="2.180642046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:00.051252863 +0000 UTC m=+848.152065577" watchObservedRunningTime="2026-09-16 13:45:00.180642046 +0000 UTC m=+848.281454763" Sep 16 13:45:00.182704 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.182673 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46ccbd-29826105-5q2dl"] Sep 16 13:45:00.185552 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.185534 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6d94e3f8-29826105-j8k55"] Sep 16 13:45:00.185688 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.185671 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46ccbd-29826105-5q2dl" Sep 16 13:45:00.187822 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.187800 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6d94e3f8-29826105-j8k55" Sep 16 13:45:00.188431 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.188414 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-thk5m\"" Sep 16 13:45:00.197066 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.197048 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46ccbd-29826105-5q2dl" Sep 16 13:45:00.204252 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.204233 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6d94e3f8-29826105-j8k55" Sep 16 13:45:00.219378 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.219352 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6d94e3f8-29826105-j8k55"] Sep 16 13:45:00.228507 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.228416 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46ccbd-29826105-5q2dl"] Sep 16 13:45:00.230397 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.230371 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-14f2a6-29826105-vjstl"] Sep 16 13:45:00.233942 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.233922 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-14f2a6-29826105-vjstl" Sep 16 13:45:00.246900 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.246875 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-14f2a6-29826105-vjstl" Sep 16 13:45:00.259975 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.259778 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-14f2a6-29826105-vjstl"] Sep 16 13:45:00.266129 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.265251 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1e5db0-29826105-tl7j6"] Sep 16 13:45:00.273414 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.273140 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1e5db0-29826105-tl7j6"] Sep 16 13:45:00.273414 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.273247 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1e5db0-29826105-tl7j6" Sep 16 13:45:00.300571 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.299260 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1e5db0-29826105-tl7j6" Sep 16 13:45:00.506910 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.506883 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46ccbd-29826105-5q2dl"] Sep 16 13:45:00.509081 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:45:00.509053 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a54a7c_7643_4ce6_a2a5_e68ee98b66f0.slice/crio-bcf7ee4f00dc01d4d5c4c83bc0275d32f6f8517ad3b5ebf6ba68cc49522c121b WatchSource:0}: Error finding container bcf7ee4f00dc01d4d5c4c83bc0275d32f6f8517ad3b5ebf6ba68cc49522c121b: Status 404 returned error can't find the container with id bcf7ee4f00dc01d4d5c4c83bc0275d32f6f8517ad3b5ebf6ba68cc49522c121b Sep 16 13:45:00.528415 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.528392 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6d94e3f8-29826105-j8k55"] Sep 16 13:45:00.537766 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:45:00.537741 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc901b4f5_d2e9_42a8_ac80_827bf1a76f6a.slice/crio-004480f51d42b84183ba126c70a2de224b6d8ad7a9fea85fcfd8cd82edebcb8c WatchSource:0}: Error finding container 004480f51d42b84183ba126c70a2de224b6d8ad7a9fea85fcfd8cd82edebcb8c: Status 404 returned error can't find the container with id 004480f51d42b84183ba126c70a2de224b6d8ad7a9fea85fcfd8cd82edebcb8c Sep 16 13:45:00.566896 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.566863 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-14f2a6-29826105-vjstl"] Sep 16 13:45:00.569879 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:45:00.569857 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a824cc3_306c_426c_ae15_e6b8a742bab0.slice/crio-c561fa51a6a8700c56541138d58b4f83e16061039ec80b44334bf23d83bcdfc7 WatchSource:0}: Error finding container c561fa51a6a8700c56541138d58b4f83e16061039ec80b44334bf23d83bcdfc7: Status 404 returned error can't find the container with id c561fa51a6a8700c56541138d58b4f83e16061039ec80b44334bf23d83bcdfc7 Sep 16 13:45:00.603522 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.603433 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1e5db0-29826105-tl7j6"] Sep 16 13:45:00.606275 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:45:00.606250 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b1b84f_c3e2_4f76_b1ab_2d9a11e69cd2.slice/crio-263bc3813ad77196ef848f51402432f94aadcbac50e7dc1543834d9650f51903 WatchSource:0}: Error finding container 263bc3813ad77196ef848f51402432f94aadcbac50e7dc1543834d9650f51903: Status 404 returned error can't find the container with id 263bc3813ad77196ef848f51402432f94aadcbac50e7dc1543834d9650f51903 Sep 16 13:45:00.625070 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:00.625041 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:01.019562 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:01.019469 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1e5db0-29826105-tl7j6" event={"ID":"b5b1b84f-c3e2-4f76-b1ab-2d9a11e69cd2","Type":"ContainerStarted","Data":"d488d9adf220157b0c83e0a457754988bbe7b68fa3c5f7408a486884d250b6c5"} Sep 16 13:45:01.019562 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:01.019533 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1e5db0-29826105-tl7j6" event={"ID":"b5b1b84f-c3e2-4f76-b1ab-2d9a11e69cd2","Type":"ContainerStarted","Data":"263bc3813ad77196ef848f51402432f94aadcbac50e7dc1543834d9650f51903"} Sep 16 13:45:01.021142 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:01.021111 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-14f2a6-29826105-vjstl" event={"ID":"1a824cc3-306c-426c-ae15-e6b8a742bab0","Type":"ContainerStarted","Data":"9394956c7cca4b97a874ba2764a3e0b6f8e68d98499483f3f912273fb6a1b6fb"} Sep 16 13:45:01.021282 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:01.021163 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-14f2a6-29826105-vjstl" event={"ID":"1a824cc3-306c-426c-ae15-e6b8a742bab0","Type":"ContainerStarted","Data":"c561fa51a6a8700c56541138d58b4f83e16061039ec80b44334bf23d83bcdfc7"} Sep 16 13:45:01.022688 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:01.022657 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6d94e3f8-29826105-j8k55" event={"ID":"c901b4f5-d2e9-42a8-ac80-827bf1a76f6a","Type":"ContainerStarted","Data":"41281cb3312cf46ffdb4cc82695f895992f60f50c240e0cd60287489639dba36"} Sep 16 13:45:01.022819 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:01.022703 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6d94e3f8-29826105-j8k55" event={"ID":"c901b4f5-d2e9-42a8-ac80-827bf1a76f6a","Type":"ContainerStarted","Data":"004480f51d42b84183ba126c70a2de224b6d8ad7a9fea85fcfd8cd82edebcb8c"} Sep 16 13:45:01.024157 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:01.024133 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46ccbd-29826105-5q2dl" event={"ID":"92a54a7c-7643-4ce6-a2a5-e68ee98b66f0","Type":"ContainerStarted","Data":"3521a0c6aa979bdfa5da0f9f349b36d76ef72fe14b1290b25b8ef63fcff00fb9"} Sep 16 13:45:01.024253 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:01.024173 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46ccbd-29826105-5q2dl" event={"ID":"92a54a7c-7643-4ce6-a2a5-e68ee98b66f0","Type":"ContainerStarted","Data":"bcf7ee4f00dc01d4d5c4c83bc0275d32f6f8517ad3b5ebf6ba68cc49522c121b"} Sep 16 13:45:01.040014 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:01.039956 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1e5db0-29826105-tl7j6" podStartSLOduration=1.039939275 podStartE2EDuration="1.039939275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:01.038495776 +0000 UTC m=+849.139308503" watchObservedRunningTime="2026-09-16 13:45:01.039939275 +0000 UTC m=+849.140751988" Sep 16 13:45:01.057301 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:01.057246 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46ccbd-29826105-5q2dl" podStartSLOduration=1.057232226 podStartE2EDuration="1.057232226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:01.055921705 +0000 UTC m=+849.156734410" watchObservedRunningTime="2026-09-16 13:45:01.057232226 +0000 UTC m=+849.158044941" Sep 16 13:45:01.075430 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:01.075365 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6d94e3f8-29826105-j8k55" podStartSLOduration=1.075344978 podStartE2EDuration="1.075344978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:01.07169728 +0000 UTC m=+849.172509995" watchObservedRunningTime="2026-09-16 13:45:01.075344978 +0000 UTC m=+849.176157693" Sep 16 13:45:01.092105 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:01.092055 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-14f2a6-29826105-vjstl" podStartSLOduration=1.0920395 podStartE2EDuration="1.0920395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:01.091080175 +0000 UTC m=+849.191892902" watchObservedRunningTime="2026-09-16 13:45:01.0920395 +0000 UTC m=+849.192852216" Sep 16 13:45:01.583816 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:01.583774 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:02.030552 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:02.030516 2580 generic.go:356] "Generic (PLEG): container finished" podID="92a54a7c-7643-4ce6-a2a5-e68ee98b66f0" containerID="3521a0c6aa979bdfa5da0f9f349b36d76ef72fe14b1290b25b8ef63fcff00fb9" exitCode=0 Sep 16 13:45:02.030993 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:02.030579 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46ccbd-29826105-5q2dl" event={"ID":"92a54a7c-7643-4ce6-a2a5-e68ee98b66f0","Type":"ContainerDied","Data":"3521a0c6aa979bdfa5da0f9f349b36d76ef72fe14b1290b25b8ef63fcff00fb9"} Sep 16 13:45:02.032193 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:02.032166 2580 generic.go:356] "Generic (PLEG): container finished" podID="b5b1b84f-c3e2-4f76-b1ab-2d9a11e69cd2" containerID="d488d9adf220157b0c83e0a457754988bbe7b68fa3c5f7408a486884d250b6c5" exitCode=0 Sep 16 13:45:02.032312 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:02.032253 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1e5db0-29826105-tl7j6" event={"ID":"b5b1b84f-c3e2-4f76-b1ab-2d9a11e69cd2","Type":"ContainerDied","Data":"d488d9adf220157b0c83e0a457754988bbe7b68fa3c5f7408a486884d250b6c5"} Sep 16 13:45:02.033728 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:02.033707 2580 generic.go:356] "Generic (PLEG): container finished" podID="1a824cc3-306c-426c-ae15-e6b8a742bab0" containerID="9394956c7cca4b97a874ba2764a3e0b6f8e68d98499483f3f912273fb6a1b6fb" exitCode=0 Sep 16 13:45:02.033857 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:02.033825 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-14f2a6-29826105-vjstl" event={"ID":"1a824cc3-306c-426c-ae15-e6b8a742bab0","Type":"ContainerDied","Data":"9394956c7cca4b97a874ba2764a3e0b6f8e68d98499483f3f912273fb6a1b6fb"} Sep 16 13:45:02.035339 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:02.035318 2580 generic.go:356] "Generic (PLEG): container finished" podID="c901b4f5-d2e9-42a8-ac80-827bf1a76f6a" containerID="41281cb3312cf46ffdb4cc82695f895992f60f50c240e0cd60287489639dba36" exitCode=7 Sep 16 13:45:02.035454 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:02.035393 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6d94e3f8-29826105-j8k55" event={"ID":"c901b4f5-d2e9-42a8-ac80-827bf1a76f6a","Type":"ContainerDied","Data":"41281cb3312cf46ffdb4cc82695f895992f60f50c240e0cd60287489639dba36"} Sep 16 13:45:02.035517 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:02.035502 2580 scope.go:117] "RemoveContainer" containerID="41281cb3312cf46ffdb4cc82695f895992f60f50c240e0cd60287489639dba36" Sep 16 13:45:03.042491 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:03.042400 2580 generic.go:356] "Generic (PLEG): container finished" podID="c901b4f5-d2e9-42a8-ac80-827bf1a76f6a" containerID="85b0dc3945246864915f231471eb0d110c5d65df59cdda9fd78ec9c1415421bc" exitCode=7 Sep 16 13:45:03.042968 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:03.042487 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6d94e3f8-29826105-j8k55" event={"ID":"c901b4f5-d2e9-42a8-ac80-827bf1a76f6a","Type":"ContainerDied","Data":"85b0dc3945246864915f231471eb0d110c5d65df59cdda9fd78ec9c1415421bc"} Sep 16 13:45:03.042968 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:03.042554 2580 scope.go:117] "RemoveContainer" containerID="41281cb3312cf46ffdb4cc82695f895992f60f50c240e0cd60287489639dba36" Sep 16 13:45:03.042968 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:03.042709 2580 scope.go:117] "RemoveContainer" containerID="85b0dc3945246864915f231471eb0d110c5d65df59cdda9fd78ec9c1415421bc" Sep 16 13:45:03.043508 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:45:03.042978 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-6d94e3f8-29826105-j8k55_odh-ai-gateway-infra(c901b4f5-d2e9-42a8-ac80-827bf1a76f6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6d94e3f8-29826105-j8k55" podUID="c901b4f5-d2e9-42a8-ac80-827bf1a76f6a" Sep 16 13:45:03.187314 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:03.187289 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-14f2a6-29826105-vjstl" Sep 16 13:45:03.215686 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:03.215663 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46ccbd-29826105-5q2dl" Sep 16 13:45:03.231853 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:03.231829 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1e5db0-29826105-tl7j6" Sep 16 13:45:04.049363 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:04.049277 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-14f2a6-29826105-vjstl" Sep 16 13:45:04.049363 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:04.049286 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-14f2a6-29826105-vjstl" event={"ID":"1a824cc3-306c-426c-ae15-e6b8a742bab0","Type":"ContainerDied","Data":"c561fa51a6a8700c56541138d58b4f83e16061039ec80b44334bf23d83bcdfc7"} Sep 16 13:45:04.049826 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:04.049399 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c561fa51a6a8700c56541138d58b4f83e16061039ec80b44334bf23d83bcdfc7" Sep 16 13:45:04.051134 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:04.051113 2580 scope.go:117] "RemoveContainer" containerID="85b0dc3945246864915f231471eb0d110c5d65df59cdda9fd78ec9c1415421bc" Sep 16 13:45:04.051395 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:45:04.051370 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-6d94e3f8-29826105-j8k55_odh-ai-gateway-infra(c901b4f5-d2e9-42a8-ac80-827bf1a76f6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6d94e3f8-29826105-j8k55" podUID="c901b4f5-d2e9-42a8-ac80-827bf1a76f6a" Sep 16 13:45:04.052277 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:04.052256 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46ccbd-29826105-5q2dl" Sep 16 13:45:04.052367 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:04.052252 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46ccbd-29826105-5q2dl" event={"ID":"92a54a7c-7643-4ce6-a2a5-e68ee98b66f0","Type":"ContainerDied","Data":"bcf7ee4f00dc01d4d5c4c83bc0275d32f6f8517ad3b5ebf6ba68cc49522c121b"} Sep 16 13:45:04.052433 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:04.052381 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcf7ee4f00dc01d4d5c4c83bc0275d32f6f8517ad3b5ebf6ba68cc49522c121b" Sep 16 13:45:04.053510 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:04.053486 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1e5db0-29826105-tl7j6" event={"ID":"b5b1b84f-c3e2-4f76-b1ab-2d9a11e69cd2","Type":"ContainerDied","Data":"263bc3813ad77196ef848f51402432f94aadcbac50e7dc1543834d9650f51903"} Sep 16 13:45:04.053619 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:04.053516 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1e5db0-29826105-tl7j6" Sep 16 13:45:04.053619 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:04.053530 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="263bc3813ad77196ef848f51402432f94aadcbac50e7dc1543834d9650f51903" Sep 16 13:45:04.374828 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:04.374745 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:11.026832 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:11.026789 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t" Sep 16 13:45:11.027239 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:11.026880 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v" Sep 16 13:45:14.306252 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:14.306223 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:45:18.355455 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:18.355424 2580 scope.go:117] "RemoveContainer" containerID="85b0dc3945246864915f231471eb0d110c5d65df59cdda9fd78ec9c1415421bc" Sep 16 13:45:19.130071 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:19.130037 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6d94e3f8-29826105-j8k55" event={"ID":"c901b4f5-d2e9-42a8-ac80-827bf1a76f6a","Type":"ContainerStarted","Data":"e0652f3ba18fb719c9704c64fb6d69e0ef34d2d1be314f5a5158a116a6ba44b7"} Sep 16 13:45:20.111048 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.111012 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:20.137555 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.137523 2580 generic.go:356] "Generic (PLEG): container finished" podID="c901b4f5-d2e9-42a8-ac80-827bf1a76f6a" containerID="e0652f3ba18fb719c9704c64fb6d69e0ef34d2d1be314f5a5158a116a6ba44b7" exitCode=0 Sep 16 13:45:20.137743 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.137594 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6d94e3f8-29826105-j8k55" event={"ID":"c901b4f5-d2e9-42a8-ac80-827bf1a76f6a","Type":"ContainerDied","Data":"e0652f3ba18fb719c9704c64fb6d69e0ef34d2d1be314f5a5158a116a6ba44b7"} Sep 16 13:45:20.137743 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.137679 2580 scope.go:117] "RemoveContainer" containerID="85b0dc3945246864915f231471eb0d110c5d65df59cdda9fd78ec9c1415421bc" Sep 16 13:45:20.155162 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.155137 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6d94e3f8-29826105-j8k55"] Sep 16 13:45:20.288233 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.288204 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr"] Sep 16 13:45:20.288642 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.288628 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a824cc3-306c-426c-ae15-e6b8a742bab0" containerName="cleanup" Sep 16 13:45:20.288686 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.288644 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a824cc3-306c-426c-ae15-e6b8a742bab0" containerName="cleanup" Sep 16 13:45:20.288686 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.288657 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92a54a7c-7643-4ce6-a2a5-e68ee98b66f0" containerName="cleanup" Sep 16 13:45:20.288686 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.288663 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="92a54a7c-7643-4ce6-a2a5-e68ee98b66f0" containerName="cleanup" Sep 16 13:45:20.288686 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.288675 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5b1b84f-c3e2-4f76-b1ab-2d9a11e69cd2" containerName="cleanup" Sep 16 13:45:20.288686 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.288680 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5b1b84f-c3e2-4f76-b1ab-2d9a11e69cd2" containerName="cleanup" Sep 16 13:45:20.288855 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.288744 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="92a54a7c-7643-4ce6-a2a5-e68ee98b66f0" containerName="cleanup" Sep 16 13:45:20.288855 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.288751 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5b1b84f-c3e2-4f76-b1ab-2d9a11e69cd2" containerName="cleanup" Sep 16 13:45:20.288855 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.288760 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a824cc3-306c-426c-ae15-e6b8a742bab0" containerName="cleanup" Sep 16 13:45:20.292875 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.292855 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:20.295653 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.295586 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-46ccbd\"/\"openshift-service-ca.crt\"" Sep 16 13:45:20.295779 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.295584 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-46ccbd\"/\"kube-root-ca.crt\"" Sep 16 13:45:20.295779 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.295743 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-46ccbd\"/\"simulator-w0-46ccbd-kserve-self-signed-certs\"" Sep 16 13:45:20.296039 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.296023 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-46ccbd\"/\"default-dockercfg-crjnl\"" Sep 16 13:45:20.303428 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.303407 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr"] Sep 16 13:45:20.324495 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.324465 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:20.338790 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.338766 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jv5k\" (UniqueName: \"kubernetes.io/projected/4471abf1-d68c-4ac9-adc0-b375ac39c347-kube-api-access-8jv5k\") pod \"simulator-w0-46ccbd-kserve-8498c66847-pwxtr\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:20.338890 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.338797 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-tmp-dir\") pod \"simulator-w0-46ccbd-kserve-8498c66847-pwxtr\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:20.338890 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.338816 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-model-cache\") pod \"simulator-w0-46ccbd-kserve-8498c66847-pwxtr\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:20.338970 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.338895 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4471abf1-d68c-4ac9-adc0-b375ac39c347-tls-certs\") pod \"simulator-w0-46ccbd-kserve-8498c66847-pwxtr\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:20.338970 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.338956 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-dshm\") pod \"simulator-w0-46ccbd-kserve-8498c66847-pwxtr\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:20.339058 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.338979 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-home\") pod \"simulator-w0-46ccbd-kserve-8498c66847-pwxtr\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:20.440314 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.440223 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-dshm\") pod \"simulator-w0-46ccbd-kserve-8498c66847-pwxtr\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:20.440314 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.440266 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-home\") pod \"simulator-w0-46ccbd-kserve-8498c66847-pwxtr\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:20.440314 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.440304 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jv5k\" (UniqueName: \"kubernetes.io/projected/4471abf1-d68c-4ac9-adc0-b375ac39c347-kube-api-access-8jv5k\") pod \"simulator-w0-46ccbd-kserve-8498c66847-pwxtr\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:20.440588 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.440322 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-tmp-dir\") pod \"simulator-w0-46ccbd-kserve-8498c66847-pwxtr\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:20.440588 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.440339 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-model-cache\") pod \"simulator-w0-46ccbd-kserve-8498c66847-pwxtr\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:20.440588 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.440362 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4471abf1-d68c-4ac9-adc0-b375ac39c347-tls-certs\") pod \"simulator-w0-46ccbd-kserve-8498c66847-pwxtr\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:20.440860 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.440806 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-home\") pod \"simulator-w0-46ccbd-kserve-8498c66847-pwxtr\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:20.440917 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.440857 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-tmp-dir\") pod \"simulator-w0-46ccbd-kserve-8498c66847-pwxtr\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:20.440917 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.440887 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-model-cache\") pod \"simulator-w0-46ccbd-kserve-8498c66847-pwxtr\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:20.442660 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.442636 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-dshm\") pod \"simulator-w0-46ccbd-kserve-8498c66847-pwxtr\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:20.442939 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.442923 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4471abf1-d68c-4ac9-adc0-b375ac39c347-tls-certs\") pod \"simulator-w0-46ccbd-kserve-8498c66847-pwxtr\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:20.455368 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.455343 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jv5k\" (UniqueName: \"kubernetes.io/projected/4471abf1-d68c-4ac9-adc0-b375ac39c347-kube-api-access-8jv5k\") pod \"simulator-w0-46ccbd-kserve-8498c66847-pwxtr\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:20.572158 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.572129 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6d94e3f8-8dgd2"] Sep 16 13:45:20.575786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.575765 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6d94e3f8-8dgd2" Sep 16 13:45:20.578397 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.578382 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:45:20.584895 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.584870 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6d94e3f8-8dgd2"] Sep 16 13:45:20.602666 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.602636 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:20.641991 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.641958 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7fff4a1-ce21-4751-8289-9fc91e53f8af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6d94e3f8-8dgd2\" (UID: \"e7fff4a1-ce21-4751-8289-9fc91e53f8af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6d94e3f8-8dgd2" Sep 16 13:45:20.743332 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.743296 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7fff4a1-ce21-4751-8289-9fc91e53f8af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6d94e3f8-8dgd2\" (UID: \"e7fff4a1-ce21-4751-8289-9fc91e53f8af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6d94e3f8-8dgd2" Sep 16 13:45:20.744026 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.743999 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7fff4a1-ce21-4751-8289-9fc91e53f8af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6d94e3f8-8dgd2\" (UID: \"e7fff4a1-ce21-4751-8289-9fc91e53f8af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6d94e3f8-8dgd2" Sep 16 13:45:20.880120 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.880094 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr"] Sep 16 13:45:20.881971 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:45:20.881940 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4471abf1_d68c_4ac9_adc0_b375ac39c347.slice/crio-a1bf21ab22351ddd88f38c48e452b15b50e21cf2cfdf9d95998fdb515b2ad6cc WatchSource:0}: Error finding container a1bf21ab22351ddd88f38c48e452b15b50e21cf2cfdf9d95998fdb515b2ad6cc: Status 404 returned error can't find the container with id a1bf21ab22351ddd88f38c48e452b15b50e21cf2cfdf9d95998fdb515b2ad6cc Sep 16 13:45:20.886877 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:20.886858 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6d94e3f8-8dgd2" Sep 16 13:45:21.146549 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:21.146514 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" event={"ID":"4471abf1-d68c-4ac9-adc0-b375ac39c347","Type":"ContainerStarted","Data":"49dc3ecf4c34e9c87c2e00f8553358b483803f412a0fea01a716f61c4eefa91d"} Sep 16 13:45:21.146977 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:21.146565 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" event={"ID":"4471abf1-d68c-4ac9-adc0-b375ac39c347","Type":"ContainerStarted","Data":"a1bf21ab22351ddd88f38c48e452b15b50e21cf2cfdf9d95998fdb515b2ad6cc"} Sep 16 13:45:21.182529 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:21.182500 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6d94e3f8-8dgd2"] Sep 16 13:45:21.183992 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:45:21.183967 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7fff4a1_ce21_4751_8289_9fc91e53f8af.slice/crio-16529257aee6ed0e2d8c36b41c22236c3cb4735ec5f6aa85b51af7f05493eacc WatchSource:0}: Error finding container 16529257aee6ed0e2d8c36b41c22236c3cb4735ec5f6aa85b51af7f05493eacc: Status 404 returned error can't find the container with id 16529257aee6ed0e2d8c36b41c22236c3cb4735ec5f6aa85b51af7f05493eacc Sep 16 13:45:21.197110 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:21.197057 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" podStartSLOduration=1.197036118 podStartE2EDuration="1.197036118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:21.195633111 +0000 UTC m=+869.296445823" watchObservedRunningTime="2026-09-16 13:45:21.197036118 +0000 UTC m=+869.297848837" Sep 16 13:45:21.266757 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:21.266733 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6d94e3f8-29826105-j8k55" Sep 16 13:45:21.338134 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:21.338101 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:22.153259 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:22.153160 2580 generic.go:356] "Generic (PLEG): container finished" podID="e7fff4a1-ce21-4751-8289-9fc91e53f8af" containerID="284a45f0641be5861ed1779a72feab41a4e1180a47ea54261a3179accc9b64fd" exitCode=0 Sep 16 13:45:22.153259 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:22.153204 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6d94e3f8-8dgd2" event={"ID":"e7fff4a1-ce21-4751-8289-9fc91e53f8af","Type":"ContainerDied","Data":"284a45f0641be5861ed1779a72feab41a4e1180a47ea54261a3179accc9b64fd"} Sep 16 13:45:22.153712 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:22.153271 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6d94e3f8-8dgd2" event={"ID":"e7fff4a1-ce21-4751-8289-9fc91e53f8af","Type":"ContainerStarted","Data":"16529257aee6ed0e2d8c36b41c22236c3cb4735ec5f6aa85b51af7f05493eacc"} Sep 16 13:45:22.154586 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:22.154564 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6d94e3f8-29826105-j8k55" Sep 16 13:45:22.154734 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:22.154629 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6d94e3f8-29826105-j8k55" event={"ID":"c901b4f5-d2e9-42a8-ac80-827bf1a76f6a","Type":"ContainerDied","Data":"004480f51d42b84183ba126c70a2de224b6d8ad7a9fea85fcfd8cd82edebcb8c"} Sep 16 13:45:22.154734 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:22.154664 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="004480f51d42b84183ba126c70a2de224b6d8ad7a9fea85fcfd8cd82edebcb8c" Sep 16 13:45:22.198031 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:22.198005 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6d94e3f8-29826105-j8k55"] Sep 16 13:45:22.212212 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:22.212185 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6d94e3f8-29826105-j8k55"] Sep 16 13:45:22.355314 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:22.355278 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c901b4f5-d2e9-42a8-ac80-827bf1a76f6a" path="/var/lib/kubelet/pods/c901b4f5-d2e9-42a8-ac80-827bf1a76f6a/volumes" Sep 16 13:45:22.360394 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:22.360370 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:23.273455 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:23.273423 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:23.287984 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:23.287962 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6d94e3f8-8dgd2" Sep 16 13:45:23.366277 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:23.366247 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7fff4a1-ce21-4751-8289-9fc91e53f8af-maas-api-service-ca\") pod \"e7fff4a1-ce21-4751-8289-9fc91e53f8af\" (UID: \"e7fff4a1-ce21-4751-8289-9fc91e53f8af\") " Sep 16 13:45:23.366714 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:23.366688 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7fff4a1-ce21-4751-8289-9fc91e53f8af-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e7fff4a1-ce21-4751-8289-9fc91e53f8af" (UID: "e7fff4a1-ce21-4751-8289-9fc91e53f8af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:45:23.467282 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:23.467255 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7fff4a1-ce21-4751-8289-9fc91e53f8af-maas-api-service-ca\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:45:24.166047 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:24.166013 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6d94e3f8-8dgd2" event={"ID":"e7fff4a1-ce21-4751-8289-9fc91e53f8af","Type":"ContainerDied","Data":"16529257aee6ed0e2d8c36b41c22236c3cb4735ec5f6aa85b51af7f05493eacc"} Sep 16 13:45:24.166047 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:24.166042 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6d94e3f8-8dgd2" Sep 16 13:45:24.166305 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:24.166063 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16529257aee6ed0e2d8c36b41c22236c3cb4735ec5f6aa85b51af7f05493eacc" Sep 16 13:45:24.186979 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:24.186950 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:25.255165 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:25.255130 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:26.024447 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:26.024414 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:26.536669 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:26.536634 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:45:27.031488 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:27.031455 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:27.923088 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:27.923047 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:28.015537 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.015504 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t"] Sep 16 13:45:28.015892 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.015861 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t" podUID="cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3" containerName="payload-processing" containerID="cri-o://af7b86e279f6f8c63d57139bc264921917c2d3ad83dd240c80a246caea83daee" gracePeriod=30 Sep 16 13:45:28.041461 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.041431 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v"] Sep 16 13:45:28.041667 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.041646 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v" podUID="e09a01f3-292d-4460-b119-1906e3d45586" containerName="payload-pre-processing" containerID="cri-o://7d999a3051a7da1f6ca0df7f3e0087aa61c66f0b9788a373314442583002e448" gracePeriod=30 Sep 16 13:45:28.163542 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.163507 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:28.190148 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.190057 2580 generic.go:356] "Generic (PLEG): container finished" podID="cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3" containerID="af7b86e279f6f8c63d57139bc264921917c2d3ad83dd240c80a246caea83daee" exitCode=0 Sep 16 13:45:28.190302 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.190233 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t" event={"ID":"cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3","Type":"ContainerDied","Data":"af7b86e279f6f8c63d57139bc264921917c2d3ad83dd240c80a246caea83daee"} Sep 16 13:45:28.191757 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.191734 2580 generic.go:356] "Generic (PLEG): container finished" podID="e09a01f3-292d-4460-b119-1906e3d45586" containerID="7d999a3051a7da1f6ca0df7f3e0087aa61c66f0b9788a373314442583002e448" exitCode=0 Sep 16 13:45:28.191876 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.191776 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v" event={"ID":"e09a01f3-292d-4460-b119-1906e3d45586","Type":"ContainerDied","Data":"7d999a3051a7da1f6ca0df7f3e0087aa61c66f0b9788a373314442583002e448"} Sep 16 13:45:28.322051 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.322025 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t" Sep 16 13:45:28.324801 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.324780 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v" Sep 16 13:45:28.356600 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.356575 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:28.363893 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.363867 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6d94e3f8-8dgd2"] Sep 16 13:45:28.365572 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.365551 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6d94e3f8-8dgd2"] Sep 16 13:45:28.412360 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.412325 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9wpx\" (UniqueName: \"kubernetes.io/projected/cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3-kube-api-access-s9wpx\") pod \"cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3\" (UID: \"cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3\") " Sep 16 13:45:28.412530 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.412369 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxdc7\" (UniqueName: \"kubernetes.io/projected/e09a01f3-292d-4460-b119-1906e3d45586-kube-api-access-vxdc7\") pod \"e09a01f3-292d-4460-b119-1906e3d45586\" (UID: \"e09a01f3-292d-4460-b119-1906e3d45586\") " Sep 16 13:45:28.412585 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.412556 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09a01f3-292d-4460-b119-1906e3d45586-plugins-config-volume\") pod \"e09a01f3-292d-4460-b119-1906e3d45586\" (UID: \"e09a01f3-292d-4460-b119-1906e3d45586\") " Sep 16 13:45:28.412664 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.412599 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3-plugins-config-volume\") pod \"cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3\" (UID: \"cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3\") " Sep 16 13:45:28.413012 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.412981 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e09a01f3-292d-4460-b119-1906e3d45586-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e09a01f3-292d-4460-b119-1906e3d45586" (UID: "e09a01f3-292d-4460-b119-1906e3d45586"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:45:28.413012 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.412981 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3" (UID: "cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:45:28.414634 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.414576 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3-kube-api-access-s9wpx" (OuterVolumeSpecName: "kube-api-access-s9wpx") pod "cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3" (UID: "cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3"). InnerVolumeSpecName "kube-api-access-s9wpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:45:28.414634 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.414620 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09a01f3-292d-4460-b119-1906e3d45586-kube-api-access-vxdc7" (OuterVolumeSpecName: "kube-api-access-vxdc7") pod "e09a01f3-292d-4460-b119-1906e3d45586" (UID: "e09a01f3-292d-4460-b119-1906e3d45586"). InnerVolumeSpecName "kube-api-access-vxdc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:45:28.459296 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.459215 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:28.514055 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.514024 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9wpx\" (UniqueName: \"kubernetes.io/projected/cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3-kube-api-access-s9wpx\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:45:28.514055 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.514052 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxdc7\" (UniqueName: \"kubernetes.io/projected/e09a01f3-292d-4460-b119-1906e3d45586-kube-api-access-vxdc7\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:45:28.514243 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.514065 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09a01f3-292d-4460-b119-1906e3d45586-plugins-config-volume\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:45:28.514243 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.514073 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3-plugins-config-volume\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:45:28.557201 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.557166 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:28.831761 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.831730 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:28.964559 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:28.964523 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:29.198543 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.198466 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v" Sep 16 13:45:29.198543 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.198498 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v" event={"ID":"e09a01f3-292d-4460-b119-1906e3d45586","Type":"ContainerDied","Data":"20e816e2b467184021172994d36b28510e66c495aa9d46e48eb1592761a6e42c"} Sep 16 13:45:29.198768 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.198559 2580 scope.go:117] "RemoveContainer" containerID="7d999a3051a7da1f6ca0df7f3e0087aa61c66f0b9788a373314442583002e448" Sep 16 13:45:29.199987 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.199970 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t" Sep 16 13:45:29.200075 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.200015 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t" event={"ID":"cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3","Type":"ContainerDied","Data":"6fec89e09f1eaf6b890ddc7dcf6007c57020d2778e4292df337cbc9095096ab4"} Sep 16 13:45:29.207591 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.207574 2580 scope.go:117] "RemoveContainer" containerID="af7b86e279f6f8c63d57139bc264921917c2d3ad83dd240c80a246caea83daee" Sep 16 13:45:29.231678 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.231657 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v"] Sep 16 13:45:29.238057 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.238033 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6d94e3f8-85c8b85fcd-qmq2v"] Sep 16 13:45:29.257658 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.254061 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t"] Sep 16 13:45:29.260875 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.260852 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6d94e3f8-6c8c8fcd86-9r48t"] Sep 16 13:45:29.301114 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.301084 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9"] Sep 16 13:45:29.301473 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.301461 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3" containerName="payload-processing" Sep 16 13:45:29.301522 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.301476 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3" containerName="payload-processing" Sep 16 13:45:29.301522 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.301484 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c901b4f5-d2e9-42a8-ac80-827bf1a76f6a" containerName="cleanup" Sep 16 13:45:29.301522 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.301489 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c901b4f5-d2e9-42a8-ac80-827bf1a76f6a" containerName="cleanup" Sep 16 13:45:29.301522 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.301505 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7fff4a1-ce21-4751-8289-9fc91e53f8af" containerName="revoke-keys" Sep 16 13:45:29.301522 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.301510 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7fff4a1-ce21-4751-8289-9fc91e53f8af" containerName="revoke-keys" Sep 16 13:45:29.301696 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.301525 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c901b4f5-d2e9-42a8-ac80-827bf1a76f6a" containerName="cleanup" Sep 16 13:45:29.301696 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.301534 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c901b4f5-d2e9-42a8-ac80-827bf1a76f6a" containerName="cleanup" Sep 16 13:45:29.301696 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.301550 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e09a01f3-292d-4460-b119-1906e3d45586" containerName="payload-pre-processing" Sep 16 13:45:29.301696 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.301558 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e09a01f3-292d-4460-b119-1906e3d45586" containerName="payload-pre-processing" Sep 16 13:45:29.301696 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.301627 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7fff4a1-ce21-4751-8289-9fc91e53f8af" containerName="revoke-keys" Sep 16 13:45:29.301696 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.301635 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3" containerName="payload-processing" Sep 16 13:45:29.301696 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.301643 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c901b4f5-d2e9-42a8-ac80-827bf1a76f6a" containerName="cleanup" Sep 16 13:45:29.301696 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.301648 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e09a01f3-292d-4460-b119-1906e3d45586" containerName="payload-pre-processing" Sep 16 13:45:29.301696 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.301655 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c901b4f5-d2e9-42a8-ac80-827bf1a76f6a" containerName="cleanup" Sep 16 13:45:29.306046 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.306031 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.311415 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.311391 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1a60997c-openshift-default-dockercfg-m2r4c\"" Sep 16 13:45:29.319174 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.319153 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9"] Sep 16 13:45:29.332426 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.332398 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:29.423403 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.423369 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istiod-ca-cert\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.423582 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.423412 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2tz7\" (UniqueName: \"kubernetes.io/projected/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-kube-api-access-z2tz7\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.423582 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.423455 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-token\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.423789 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.423595 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-workload-socket\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.423789 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.423651 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-envoy\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.423789 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.423709 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-workload-certs\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.423789 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.423725 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-podinfo\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.423789 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.423744 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-credential-socket\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.423985 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.423807 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-data\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.525244 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.525208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istiod-ca-cert\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.525415 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.525258 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2tz7\" (UniqueName: \"kubernetes.io/projected/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-kube-api-access-z2tz7\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.525415 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.525310 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-token\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.525551 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.525452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-workload-socket\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.525551 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.525496 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-envoy\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.525689 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.525556 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-workload-certs\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.525689 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.525580 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-podinfo\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.525794 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.525749 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-credential-socket\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.525794 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.525787 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-data\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.525896 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.525825 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-workload-socket\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.525983 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.525963 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-workload-certs\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.526045 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.526008 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istiod-ca-cert\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.526184 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.526157 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-credential-socket\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.526229 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.526171 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-data\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.527868 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.527850 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-envoy\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.528248 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.528225 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-podinfo\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.537336 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.537307 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-token\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.537719 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.537703 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2tz7\" (UniqueName: \"kubernetes.io/projected/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-kube-api-access-z2tz7\") pod \"e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.616309 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.616278 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:29.650763 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.650730 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:29.898800 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.898769 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9"] Sep 16 13:45:29.902192 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:45:29.902168 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7c901a_2b97_4762_b1cf_3feaf6936dbc.slice/crio-a070253bd9768f6190819d089f9b10e1aa31a1f2850b99eeaba65ad718d30712 WatchSource:0}: Error finding container a070253bd9768f6190819d089f9b10e1aa31a1f2850b99eeaba65ad718d30712: Status 404 returned error can't find the container with id a070253bd9768f6190819d089f9b10e1aa31a1f2850b99eeaba65ad718d30712 Sep 16 13:45:29.904920 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.904883 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:45:29.905022 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.904954 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:45:29.905022 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.904983 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:45:29.987580 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:29.987553 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:30.208465 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:30.208355 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" event={"ID":"5b7c901a-2b97-4762-b1cf-3feaf6936dbc","Type":"ContainerStarted","Data":"92f1ab13a964bfb0d60a3f37be3c1c79ef0c2b159509f78d142217d968f7337d"} Sep 16 13:45:30.208465 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:30.208423 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" event={"ID":"5b7c901a-2b97-4762-b1cf-3feaf6936dbc","Type":"ContainerStarted","Data":"a070253bd9768f6190819d089f9b10e1aa31a1f2850b99eeaba65ad718d30712"} Sep 16 13:45:30.240918 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:30.240865 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" podStartSLOduration=1.2408481980000001 podStartE2EDuration="1.240848198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:30.238409565 +0000 UTC m=+878.339222280" watchObservedRunningTime="2026-09-16 13:45:30.240848198 +0000 UTC m=+878.341660913" Sep 16 13:45:30.356884 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:30.356836 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3" path="/var/lib/kubelet/pods/cc69ec3d-b78c-4073-ae1a-c9bd2a13fca3/volumes" Sep 16 13:45:30.357243 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:30.357228 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e09a01f3-292d-4460-b119-1906e3d45586" path="/var/lib/kubelet/pods/e09a01f3-292d-4460-b119-1906e3d45586/volumes" Sep 16 13:45:30.357600 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:30.357582 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7fff4a1-ce21-4751-8289-9fc91e53f8af" path="/var/lib/kubelet/pods/e7fff4a1-ce21-4751-8289-9fc91e53f8af/volumes" Sep 16 13:45:30.490369 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:30.490340 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:30.603063 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:30.603028 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:30.603230 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:30.603079 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:30.615911 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:30.615886 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:30.616972 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:30.616955 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:30.622011 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:30.621990 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:31.214701 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:31.214666 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:31.215800 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:31.215781 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:31.225792 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:31.225769 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:45:31.432531 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:31.432496 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:31.637276 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:31.637199 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:31.738339 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:31.738310 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:32.016328 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.016291 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88"] Sep 16 13:45:32.016697 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.016682 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c901b4f5-d2e9-42a8-ac80-827bf1a76f6a" containerName="cleanup" Sep 16 13:45:32.016741 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.016702 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c901b4f5-d2e9-42a8-ac80-827bf1a76f6a" containerName="cleanup" Sep 16 13:45:32.016791 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.016780 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c901b4f5-d2e9-42a8-ac80-827bf1a76f6a" containerName="cleanup" Sep 16 13:45:32.021037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.021017 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:32.025789 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.025770 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4bb29c\"/\"default-dockercfg-84r6h\"" Sep 16 13:45:32.026042 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.026028 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4bb29c\"/\"openshift-service-ca.crt\"" Sep 16 13:45:32.026836 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.026818 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4bb29c\"/\"simulator-w4-4bb29c-kserve-self-signed-certs\"" Sep 16 13:45:32.031034 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.031015 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4bb29c\"/\"kube-root-ca.crt\"" Sep 16 13:45:32.038644 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.038620 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88"] Sep 16 13:45:32.149235 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.149202 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-dshm\") pod \"simulator-w4-4bb29c-kserve-5f79546d7-f2w88\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:32.149399 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.149243 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-home\") pod \"simulator-w4-4bb29c-kserve-5f79546d7-f2w88\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:32.149399 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.149365 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-tmp-dir\") pod \"simulator-w4-4bb29c-kserve-5f79546d7-f2w88\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:32.149481 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.149399 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnrvl\" (UniqueName: \"kubernetes.io/projected/3b14e93f-fcba-4664-bbd9-98475f21b719-kube-api-access-qnrvl\") pod \"simulator-w4-4bb29c-kserve-5f79546d7-f2w88\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:32.149481 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.149444 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b14e93f-fcba-4664-bbd9-98475f21b719-tls-certs\") pod \"simulator-w4-4bb29c-kserve-5f79546d7-f2w88\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:32.149481 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.149469 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-model-cache\") pod \"simulator-w4-4bb29c-kserve-5f79546d7-f2w88\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:32.250331 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.250295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-tmp-dir\") pod \"simulator-w4-4bb29c-kserve-5f79546d7-f2w88\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:32.250331 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.250331 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnrvl\" (UniqueName: \"kubernetes.io/projected/3b14e93f-fcba-4664-bbd9-98475f21b719-kube-api-access-qnrvl\") pod \"simulator-w4-4bb29c-kserve-5f79546d7-f2w88\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:32.250867 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.250365 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b14e93f-fcba-4664-bbd9-98475f21b719-tls-certs\") pod \"simulator-w4-4bb29c-kserve-5f79546d7-f2w88\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:32.250867 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.250391 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-model-cache\") pod \"simulator-w4-4bb29c-kserve-5f79546d7-f2w88\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:32.250867 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.250429 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-dshm\") pod \"simulator-w4-4bb29c-kserve-5f79546d7-f2w88\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:32.250867 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.250467 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-home\") pod \"simulator-w4-4bb29c-kserve-5f79546d7-f2w88\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:32.250867 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.250797 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-tmp-dir\") pod \"simulator-w4-4bb29c-kserve-5f79546d7-f2w88\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:32.250867 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.250834 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-model-cache\") pod \"simulator-w4-4bb29c-kserve-5f79546d7-f2w88\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:32.251085 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.250879 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-home\") pod \"simulator-w4-4bb29c-kserve-5f79546d7-f2w88\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:32.252800 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.252779 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-dshm\") pod \"simulator-w4-4bb29c-kserve-5f79546d7-f2w88\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:32.253073 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.253053 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b14e93f-fcba-4664-bbd9-98475f21b719-tls-certs\") pod \"simulator-w4-4bb29c-kserve-5f79546d7-f2w88\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:32.275910 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.275844 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnrvl\" (UniqueName: \"kubernetes.io/projected/3b14e93f-fcba-4664-bbd9-98475f21b719-kube-api-access-qnrvl\") pod \"simulator-w4-4bb29c-kserve-5f79546d7-f2w88\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:32.330581 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.330544 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:32.397719 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.397681 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:32.613113 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.613088 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88"] Sep 16 13:45:32.614924 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:45:32.614892 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b14e93f_fcba_4664_bbd9_98475f21b719.slice/crio-29e58bd2ec1b366cc02b9ff36d1aa4fd14fe1e108a923a672216ab72557c263a WatchSource:0}: Error finding container 29e58bd2ec1b366cc02b9ff36d1aa4fd14fe1e108a923a672216ab72557c263a: Status 404 returned error can't find the container with id 29e58bd2ec1b366cc02b9ff36d1aa4fd14fe1e108a923a672216ab72557c263a Sep 16 13:45:32.781024 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:32.780990 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:33.226952 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:33.226908 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" event={"ID":"3b14e93f-fcba-4664-bbd9-98475f21b719","Type":"ContainerStarted","Data":"e5ff4bae40f58a128ad9497c9eb71439c040c7dabc1aec5a9bd0e138c8fde4fa"} Sep 16 13:45:33.227165 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:33.226960 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" event={"ID":"3b14e93f-fcba-4664-bbd9-98475f21b719","Type":"ContainerStarted","Data":"29e58bd2ec1b366cc02b9ff36d1aa4fd14fe1e108a923a672216ab72557c263a"} Sep 16 13:45:33.247910 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:33.247856 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" podStartSLOduration=2.247840229 podStartE2EDuration="2.247840229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:33.246213769 +0000 UTC m=+881.347026497" watchObservedRunningTime="2026-09-16 13:45:33.247840229 +0000 UTC m=+881.348652992" Sep 16 13:45:33.638370 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:33.638288 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:34.477789 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:34.477747 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:34.579027 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:34.578993 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:34.911048 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:34.910963 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:35.102439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:35.102411 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:35.201407 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:35.201320 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:35.306010 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:35.305972 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn"] Sep 16 13:45:35.309699 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:35.309681 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn" Sep 16 13:45:35.316321 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:35.316282 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1a60997c-dockercfg-wg8kl\"" Sep 16 13:45:35.316455 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:35.316369 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1a60997c\"" Sep 16 13:45:35.325908 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:35.325882 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn"] Sep 16 13:45:35.378818 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:35.378777 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0a207d3-89da-4c7b-9601-d14c289360b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn\" (UID: \"f0a207d3-89da-4c7b-9601-d14c289360b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn" Sep 16 13:45:35.379001 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:35.378838 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kpf6\" (UniqueName: \"kubernetes.io/projected/f0a207d3-89da-4c7b-9601-d14c289360b4-kube-api-access-9kpf6\") pod \"payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn\" (UID: \"f0a207d3-89da-4c7b-9601-d14c289360b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn" Sep 16 13:45:35.479732 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:35.479698 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kpf6\" (UniqueName: \"kubernetes.io/projected/f0a207d3-89da-4c7b-9601-d14c289360b4-kube-api-access-9kpf6\") pod \"payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn\" (UID: \"f0a207d3-89da-4c7b-9601-d14c289360b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn" Sep 16 13:45:35.479891 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:35.479814 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0a207d3-89da-4c7b-9601-d14c289360b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn\" (UID: \"f0a207d3-89da-4c7b-9601-d14c289360b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn" Sep 16 13:45:35.480426 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:35.480408 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0a207d3-89da-4c7b-9601-d14c289360b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn\" (UID: \"f0a207d3-89da-4c7b-9601-d14c289360b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn" Sep 16 13:45:35.488599 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:35.488573 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kpf6\" (UniqueName: \"kubernetes.io/projected/f0a207d3-89da-4c7b-9601-d14c289360b4-kube-api-access-9kpf6\") pod \"payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn\" (UID: \"f0a207d3-89da-4c7b-9601-d14c289360b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn" Sep 16 13:45:35.579784 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:35.579753 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:35.620553 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:35.620525 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn" Sep 16 13:45:35.915905 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:45:35.915875 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a207d3_89da_4c7b_9601_d14c289360b4.slice/crio-5b86d1bc3ea8dfc0ce1b287ac25777eafa7b958cb6098c0190cf37ebe6b7a5cd WatchSource:0}: Error finding container 5b86d1bc3ea8dfc0ce1b287ac25777eafa7b958cb6098c0190cf37ebe6b7a5cd: Status 404 returned error can't find the container with id 5b86d1bc3ea8dfc0ce1b287ac25777eafa7b958cb6098c0190cf37ebe6b7a5cd Sep 16 13:45:35.926695 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:35.926663 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn"] Sep 16 13:45:36.246224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:36.246192 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn" event={"ID":"f0a207d3-89da-4c7b-9601-d14c289360b4","Type":"ContainerStarted","Data":"977daa3b7401dbae4cb04ecfc4fcf4b86d3519c61bcac6164af4fb8689ac3cb2"} Sep 16 13:45:36.246432 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:36.246243 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn" event={"ID":"f0a207d3-89da-4c7b-9601-d14c289360b4","Type":"ContainerStarted","Data":"5b86d1bc3ea8dfc0ce1b287ac25777eafa7b958cb6098c0190cf37ebe6b7a5cd"} Sep 16 13:45:36.246432 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:36.246327 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn" Sep 16 13:45:36.271286 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:36.271240 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn" podStartSLOduration=1.271226124 podStartE2EDuration="1.271226124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:36.270977535 +0000 UTC m=+884.371790251" watchObservedRunningTime="2026-09-16 13:45:36.271226124 +0000 UTC m=+884.372038838" Sep 16 13:45:36.741823 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:36.741782 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:37.855449 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:37.855412 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:38.644882 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:38.644847 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:42.331450 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:42.331410 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:42.331919 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:42.331470 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:42.344657 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:42.344633 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:43.291911 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:43.291874 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:45:45.724334 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:45.724301 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:45:47.253125 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:47.253096 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn" Sep 16 13:45:50.594955 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.594920 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t"] Sep 16 13:45:50.598388 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.598368 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:45:50.601275 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.601231 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-14f2a6\"/\"openshift-service-ca.crt\"" Sep 16 13:45:50.602913 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.602891 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-14f2a6\"/\"default-dockercfg-b7t8v\"" Sep 16 13:45:50.603144 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.602957 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-14f2a6\"/\"kube-root-ca.crt\"" Sep 16 13:45:50.603753 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.603732 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-14f2a6\"/\"autores-model-14f2a6-kserve-self-signed-certs\"" Sep 16 13:45:50.609966 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.609944 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t"] Sep 16 13:45:50.612975 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.612953 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:50.733507 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.733471 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-tls-certs\") pod \"autores-model-14f2a6-kserve-84c5b797dc-8g59t\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:45:50.733711 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.733545 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-home\") pod \"autores-model-14f2a6-kserve-84c5b797dc-8g59t\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:45:50.733711 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.733633 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-tmp-dir\") pod \"autores-model-14f2a6-kserve-84c5b797dc-8g59t\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:45:50.733711 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.733672 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz6pw\" (UniqueName: \"kubernetes.io/projected/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-kube-api-access-kz6pw\") pod \"autores-model-14f2a6-kserve-84c5b797dc-8g59t\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:45:50.733857 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.733807 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-dshm\") pod \"autores-model-14f2a6-kserve-84c5b797dc-8g59t\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:45:50.733991 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.733970 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-model-cache\") pod \"autores-model-14f2a6-kserve-84c5b797dc-8g59t\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:45:50.834969 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.834935 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-dshm\") pod \"autores-model-14f2a6-kserve-84c5b797dc-8g59t\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:45:50.835148 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.835030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-model-cache\") pod \"autores-model-14f2a6-kserve-84c5b797dc-8g59t\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:45:50.835148 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.835064 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-tls-certs\") pod \"autores-model-14f2a6-kserve-84c5b797dc-8g59t\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:45:50.835148 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.835122 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-home\") pod \"autores-model-14f2a6-kserve-84c5b797dc-8g59t\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:45:50.835148 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.835143 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-tmp-dir\") pod \"autores-model-14f2a6-kserve-84c5b797dc-8g59t\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:45:50.835312 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.835167 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz6pw\" (UniqueName: \"kubernetes.io/projected/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-kube-api-access-kz6pw\") pod \"autores-model-14f2a6-kserve-84c5b797dc-8g59t\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:45:50.835472 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.835446 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-model-cache\") pod \"autores-model-14f2a6-kserve-84c5b797dc-8g59t\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:45:50.835595 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.835506 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-home\") pod \"autores-model-14f2a6-kserve-84c5b797dc-8g59t\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:45:50.835595 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.835550 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-tmp-dir\") pod \"autores-model-14f2a6-kserve-84c5b797dc-8g59t\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:45:50.837411 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.837394 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-dshm\") pod \"autores-model-14f2a6-kserve-84c5b797dc-8g59t\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:45:50.837681 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.837666 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-tls-certs\") pod \"autores-model-14f2a6-kserve-84c5b797dc-8g59t\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:45:50.843900 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.843883 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz6pw\" (UniqueName: \"kubernetes.io/projected/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-kube-api-access-kz6pw\") pod \"autores-model-14f2a6-kserve-84c5b797dc-8g59t\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:45:50.911224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:50.911128 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:45:51.215236 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:51.215207 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t"] Sep 16 13:45:51.216425 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:45:51.216401 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fc4f13d_201c_46ec_bf3a_be2c2b54b6ee.slice/crio-238e35240271caa6a35eb91ccb912145a8e7c23ede69fa0e362018b0de059967 WatchSource:0}: Error finding container 238e35240271caa6a35eb91ccb912145a8e7c23ede69fa0e362018b0de059967: Status 404 returned error can't find the container with id 238e35240271caa6a35eb91ccb912145a8e7c23ede69fa0e362018b0de059967 Sep 16 13:45:51.330091 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:51.330051 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" event={"ID":"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee","Type":"ContainerStarted","Data":"07a69cb29b0b2e208e465d0b816afff15770924aecfcf7321ca3f45f4fc643d5"} Sep 16 13:45:51.330224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:51.330105 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" event={"ID":"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee","Type":"ContainerStarted","Data":"238e35240271caa6a35eb91ccb912145a8e7c23ede69fa0e362018b0de059967"} Sep 16 13:45:51.351202 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:51.351152 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" podStartSLOduration=1.351135614 podStartE2EDuration="1.351135614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:51.349383859 +0000 UTC m=+899.450196573" watchObservedRunningTime="2026-09-16 13:45:51.351135614 +0000 UTC m=+899.451948332" Sep 16 13:45:53.233727 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:53.233686 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:55.729884 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:55.729851 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:56.357645 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:56.357596 2580 generic.go:356] "Generic (PLEG): container finished" podID="5b7c901a-2b97-4762-b1cf-3feaf6936dbc" containerID="92f1ab13a964bfb0d60a3f37be3c1c79ef0c2b159509f78d142217d968f7337d" exitCode=0 Sep 16 13:45:56.357829 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:56.357674 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" event={"ID":"5b7c901a-2b97-4762-b1cf-3feaf6936dbc","Type":"ContainerDied","Data":"92f1ab13a964bfb0d60a3f37be3c1c79ef0c2b159509f78d142217d968f7337d"} Sep 16 13:45:56.358060 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:56.358047 2580 scope.go:117] "RemoveContainer" containerID="92f1ab13a964bfb0d60a3f37be3c1c79ef0c2b159509f78d142217d968f7337d" Sep 16 13:45:56.359329 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:56.359299 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:45:56.359391 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:56.359369 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:45:56.359426 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:56.359400 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:45:57.203275 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:57.203247 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:57.368182 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:57.368135 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" event={"ID":"5b7c901a-2b97-4762-b1cf-3feaf6936dbc","Type":"ContainerStarted","Data":"33c1e9ef8a20307bbb52acb785a5389fa998695d0a115eb0d2795aa867452f08"} Sep 16 13:45:57.617091 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:57.617049 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:45:57.618242 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:57.618219 2580 patch_prober.go:28] interesting pod/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 16 13:45:57.618302 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:57.618283 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" podUID="5b7c901a-2b97-4762-b1cf-3feaf6936dbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 16 13:45:57.876863 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:57.876756 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:58.616761 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:58.616717 2580 patch_prober.go:28] interesting pod/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 16 13:45:58.617231 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:58.616780 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" podUID="5b7c901a-2b97-4762-b1cf-3feaf6936dbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 16 13:45:58.822401 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:58.822361 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:59.616815 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:59.616776 2580 patch_prober.go:28] interesting pod/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 16 13:45:59.617211 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:59.616843 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" podUID="5b7c901a-2b97-4762-b1cf-3feaf6936dbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 16 13:45:59.720148 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:59.720109 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:59.721756 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:45:59.721737 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:46:00.607360 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:00.607327 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:00.617281 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:00.617255 2580 patch_prober.go:28] interesting pod/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 16 13:46:00.617674 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:00.617310 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" podUID="5b7c901a-2b97-4762-b1cf-3feaf6936dbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 16 13:46:00.911711 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:00.911586 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:46:00.911711 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:00.911680 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:46:00.924140 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:00.924113 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:46:01.215407 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.215355 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:46:01.376353 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.376218 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:01.409293 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.409265 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:46:01.505457 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.505371 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr"] Sep 16 13:46:01.507871 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.507853 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:01.511291 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.511268 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-aca190\"/\"kube-root-ca.crt\"" Sep 16 13:46:01.511427 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.511362 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-aca190\"/\"test-model-aca190-kserve-self-signed-certs\"" Sep 16 13:46:01.511427 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.511398 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-aca190\"/\"openshift-service-ca.crt\"" Sep 16 13:46:01.511427 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.511419 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-aca190\"/\"default-dockercfg-bcmjb\"" Sep 16 13:46:01.523924 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.523897 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr"] Sep 16 13:46:01.536635 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.536593 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-model-cache\") pod \"test-model-aca190-kserve-7cdbb86b75-bq5xr\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:01.536802 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.536671 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-dshm\") pod \"test-model-aca190-kserve-7cdbb86b75-bq5xr\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:01.536802 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.536741 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bfccddd-ec33-40ed-a817-e7be2dc39964-tls-certs\") pod \"test-model-aca190-kserve-7cdbb86b75-bq5xr\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:01.536802 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.536790 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-home\") pod \"test-model-aca190-kserve-7cdbb86b75-bq5xr\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:01.536965 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.536869 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l87nw\" (UniqueName: \"kubernetes.io/projected/0bfccddd-ec33-40ed-a817-e7be2dc39964-kube-api-access-l87nw\") pod \"test-model-aca190-kserve-7cdbb86b75-bq5xr\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:01.536965 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.536958 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-tmp-dir\") pod \"test-model-aca190-kserve-7cdbb86b75-bq5xr\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:01.617399 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.617363 2580 patch_prober.go:28] interesting pod/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 16 13:46:01.617871 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.617425 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" podUID="5b7c901a-2b97-4762-b1cf-3feaf6936dbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 16 13:46:01.638250 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.638217 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-tmp-dir\") pod \"test-model-aca190-kserve-7cdbb86b75-bq5xr\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:01.638421 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.638269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-model-cache\") pod \"test-model-aca190-kserve-7cdbb86b75-bq5xr\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:01.638421 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.638296 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-dshm\") pod \"test-model-aca190-kserve-7cdbb86b75-bq5xr\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:01.638421 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.638324 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bfccddd-ec33-40ed-a817-e7be2dc39964-tls-certs\") pod \"test-model-aca190-kserve-7cdbb86b75-bq5xr\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:01.638421 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.638356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-home\") pod \"test-model-aca190-kserve-7cdbb86b75-bq5xr\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:01.638421 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.638418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l87nw\" (UniqueName: \"kubernetes.io/projected/0bfccddd-ec33-40ed-a817-e7be2dc39964-kube-api-access-l87nw\") pod \"test-model-aca190-kserve-7cdbb86b75-bq5xr\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:01.638718 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.638667 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-model-cache\") pod \"test-model-aca190-kserve-7cdbb86b75-bq5xr\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:01.638718 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.638666 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-tmp-dir\") pod \"test-model-aca190-kserve-7cdbb86b75-bq5xr\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:01.638819 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.638802 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-home\") pod \"test-model-aca190-kserve-7cdbb86b75-bq5xr\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:01.640006 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.639982 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:01.640941 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.640917 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-dshm\") pod \"test-model-aca190-kserve-7cdbb86b75-bq5xr\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:01.641120 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.641103 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bfccddd-ec33-40ed-a817-e7be2dc39964-tls-certs\") pod \"test-model-aca190-kserve-7cdbb86b75-bq5xr\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:01.652997 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.652966 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l87nw\" (UniqueName: \"kubernetes.io/projected/0bfccddd-ec33-40ed-a817-e7be2dc39964-kube-api-access-l87nw\") pod \"test-model-aca190-kserve-7cdbb86b75-bq5xr\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:01.817040 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:01.816940 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:02.124842 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:02.124816 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr"] Sep 16 13:46:02.127531 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:46:02.127504 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bfccddd_ec33_40ed_a817_e7be2dc39964.slice/crio-12787059bbba730da244fe6f4b59e2e71373dd467a42bebe6ae65189e3920524 WatchSource:0}: Error finding container 12787059bbba730da244fe6f4b59e2e71373dd467a42bebe6ae65189e3920524: Status 404 returned error can't find the container with id 12787059bbba730da244fe6f4b59e2e71373dd467a42bebe6ae65189e3920524 Sep 16 13:46:02.333618 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:02.333576 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:02.406224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:02.406104 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" event={"ID":"0bfccddd-ec33-40ed-a817-e7be2dc39964","Type":"ContainerStarted","Data":"c15aa8d758d2f56d9a76068a4a139223beeebdbbade73403dbc4bdaf0c1a42fd"} Sep 16 13:46:02.406224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:02.406160 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" event={"ID":"0bfccddd-ec33-40ed-a817-e7be2dc39964","Type":"ContainerStarted","Data":"12787059bbba730da244fe6f4b59e2e71373dd467a42bebe6ae65189e3920524"} Sep 16 13:46:02.437700 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:02.437648 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" podStartSLOduration=1.437631009 podStartE2EDuration="1.437631009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:46:02.435079459 +0000 UTC m=+910.535892175" watchObservedRunningTime="2026-09-16 13:46:02.437631009 +0000 UTC m=+910.538443718" Sep 16 13:46:02.620979 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:02.620951 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:46:02.622050 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:02.622032 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:46:03.545025 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:03.544991 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:04.559450 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:04.559410 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:05.288224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:05.288189 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:06.151795 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:06.151765 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:07.053121 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:07.053084 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:07.732406 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:07.732379 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:08.832832 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:08.832796 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:11.817685 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:11.817651 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:11.818056 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:11.817937 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:11.830159 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:11.830137 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:12.468592 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:12.468560 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:46:13.483226 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:13.483171 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:13.521672 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:13.521638 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn"] Sep 16 13:46:13.521911 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:13.521882 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn" podUID="f0a207d3-89da-4c7b-9601-d14c289360b4" containerName="payload-pre-processing" containerID="cri-o://977daa3b7401dbae4cb04ecfc4fcf4b86d3519c61bcac6164af4fb8689ac3cb2" gracePeriod=30 Sep 16 13:46:13.583534 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:13.583508 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:13.682973 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:13.682944 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:13.777124 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:13.777096 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn" Sep 16 13:46:13.783250 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:13.783227 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:13.856687 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:13.856651 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0a207d3-89da-4c7b-9601-d14c289360b4-plugins-config-volume\") pod \"f0a207d3-89da-4c7b-9601-d14c289360b4\" (UID: \"f0a207d3-89da-4c7b-9601-d14c289360b4\") " Sep 16 13:46:13.856856 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:13.856749 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kpf6\" (UniqueName: \"kubernetes.io/projected/f0a207d3-89da-4c7b-9601-d14c289360b4-kube-api-access-9kpf6\") pod \"f0a207d3-89da-4c7b-9601-d14c289360b4\" (UID: \"f0a207d3-89da-4c7b-9601-d14c289360b4\") " Sep 16 13:46:13.857039 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:13.857017 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a207d3-89da-4c7b-9601-d14c289360b4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f0a207d3-89da-4c7b-9601-d14c289360b4" (UID: "f0a207d3-89da-4c7b-9601-d14c289360b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:46:13.859012 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:13.858994 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a207d3-89da-4c7b-9601-d14c289360b4-kube-api-access-9kpf6" (OuterVolumeSpecName: "kube-api-access-9kpf6") pod "f0a207d3-89da-4c7b-9601-d14c289360b4" (UID: "f0a207d3-89da-4c7b-9601-d14c289360b4"). InnerVolumeSpecName "kube-api-access-9kpf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:46:13.958332 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:13.958291 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0a207d3-89da-4c7b-9601-d14c289360b4-plugins-config-volume\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:46:13.958332 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:13.958329 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kpf6\" (UniqueName: \"kubernetes.io/projected/f0a207d3-89da-4c7b-9601-d14c289360b4-kube-api-access-9kpf6\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:46:14.167544 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:14.167451 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:14.468760 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:14.468726 2580 generic.go:356] "Generic (PLEG): container finished" podID="f0a207d3-89da-4c7b-9601-d14c289360b4" containerID="977daa3b7401dbae4cb04ecfc4fcf4b86d3519c61bcac6164af4fb8689ac3cb2" exitCode=0 Sep 16 13:46:14.468934 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:14.468770 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn" Sep 16 13:46:14.468934 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:14.468804 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn" event={"ID":"f0a207d3-89da-4c7b-9601-d14c289360b4","Type":"ContainerDied","Data":"977daa3b7401dbae4cb04ecfc4fcf4b86d3519c61bcac6164af4fb8689ac3cb2"} Sep 16 13:46:14.468934 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:14.468857 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn" event={"ID":"f0a207d3-89da-4c7b-9601-d14c289360b4","Type":"ContainerDied","Data":"5b86d1bc3ea8dfc0ce1b287ac25777eafa7b958cb6098c0190cf37ebe6b7a5cd"} Sep 16 13:46:14.468934 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:14.468887 2580 scope.go:117] "RemoveContainer" containerID="977daa3b7401dbae4cb04ecfc4fcf4b86d3519c61bcac6164af4fb8689ac3cb2" Sep 16 13:46:14.485763 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:14.485744 2580 scope.go:117] "RemoveContainer" containerID="977daa3b7401dbae4cb04ecfc4fcf4b86d3519c61bcac6164af4fb8689ac3cb2" Sep 16 13:46:14.486180 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:46:14.486149 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"977daa3b7401dbae4cb04ecfc4fcf4b86d3519c61bcac6164af4fb8689ac3cb2\": container with ID starting with 977daa3b7401dbae4cb04ecfc4fcf4b86d3519c61bcac6164af4fb8689ac3cb2 not found: ID does not exist" containerID="977daa3b7401dbae4cb04ecfc4fcf4b86d3519c61bcac6164af4fb8689ac3cb2" Sep 16 13:46:14.486281 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:14.486189 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"977daa3b7401dbae4cb04ecfc4fcf4b86d3519c61bcac6164af4fb8689ac3cb2"} err="failed to get container status \"977daa3b7401dbae4cb04ecfc4fcf4b86d3519c61bcac6164af4fb8689ac3cb2\": rpc error: code = NotFound desc = could not find container \"977daa3b7401dbae4cb04ecfc4fcf4b86d3519c61bcac6164af4fb8689ac3cb2\": container with ID starting with 977daa3b7401dbae4cb04ecfc4fcf4b86d3519c61bcac6164af4fb8689ac3cb2 not found: ID does not exist" Sep 16 13:46:14.488807 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:14.488783 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn"] Sep 16 13:46:14.492665 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:14.492643 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a60997c-fcc6886f4-cf6qn"] Sep 16 13:46:14.945794 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:14.945708 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:15.006541 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:15.006501 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9"] Sep 16 13:46:15.006794 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:15.006753 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" podUID="5b7c901a-2b97-4762-b1cf-3feaf6936dbc" containerName="istio-proxy" containerID="cri-o://33c1e9ef8a20307bbb52acb785a5389fa998695d0a115eb0d2795aa867452f08" gracePeriod=30 Sep 16 13:46:15.059773 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:15.059742 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:15.565143 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:15.565104 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:15.905011 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:15.904924 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:16.070722 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:16.070679 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:16.172126 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:16.172031 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:16.355700 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:16.355658 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0a207d3-89da-4c7b-9601-d14c289360b4" path="/var/lib/kubelet/pods/f0a207d3-89da-4c7b-9601-d14c289360b4/volumes" Sep 16 13:46:16.935242 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:16.935202 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:17.624395 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:17.624355 2580 patch_prober.go:28] interesting pod/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 13:46:17.624595 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:17.624427 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" podUID="5b7c901a-2b97-4762-b1cf-3feaf6936dbc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 13:46:17.833130 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:17.833095 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:18.738537 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:18.738506 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:18.863650 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:18.863581 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:18.974730 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:18.974700 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:19.735896 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:19.735861 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:20.271124 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.271063 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:46:20.424231 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.424190 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-podinfo\") pod \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " Sep 16 13:46:20.424415 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.424262 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-workload-certs\") pod \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " Sep 16 13:46:20.424415 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.424295 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-token\") pod \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " Sep 16 13:46:20.424415 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.424321 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-envoy\") pod \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " Sep 16 13:46:20.424415 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.424363 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-workload-socket\") pod \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " Sep 16 13:46:20.424415 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.424395 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-data\") pod \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " Sep 16 13:46:20.424720 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.424426 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-credential-socket\") pod \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " Sep 16 13:46:20.424720 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.424456 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2tz7\" (UniqueName: \"kubernetes.io/projected/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-kube-api-access-z2tz7\") pod \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " Sep 16 13:46:20.424720 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.424485 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istiod-ca-cert\") pod \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\" (UID: \"5b7c901a-2b97-4762-b1cf-3feaf6936dbc\") " Sep 16 13:46:20.424720 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.424644 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5b7c901a-2b97-4762-b1cf-3feaf6936dbc" (UID: "5b7c901a-2b97-4762-b1cf-3feaf6936dbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:46:20.424720 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.424690 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-data" (OuterVolumeSpecName: "istio-data") pod "5b7c901a-2b97-4762-b1cf-3feaf6936dbc" (UID: "5b7c901a-2b97-4762-b1cf-3feaf6936dbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:46:20.424983 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.424752 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5b7c901a-2b97-4762-b1cf-3feaf6936dbc" (UID: "5b7c901a-2b97-4762-b1cf-3feaf6936dbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:46:20.424983 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.424868 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5b7c901a-2b97-4762-b1cf-3feaf6936dbc" (UID: "5b7c901a-2b97-4762-b1cf-3feaf6936dbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:46:20.424983 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.424943 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5b7c901a-2b97-4762-b1cf-3feaf6936dbc" (UID: "5b7c901a-2b97-4762-b1cf-3feaf6936dbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:46:20.425234 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.425017 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-workload-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:46:20.425234 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.425032 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-workload-socket\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:46:20.425234 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.425040 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-data\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:46:20.425234 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.425048 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-credential-socket\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:46:20.426784 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.426756 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5b7c901a-2b97-4762-b1cf-3feaf6936dbc" (UID: "5b7c901a-2b97-4762-b1cf-3feaf6936dbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:46:20.427026 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.427002 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-kube-api-access-z2tz7" (OuterVolumeSpecName: "kube-api-access-z2tz7") pod "5b7c901a-2b97-4762-b1cf-3feaf6936dbc" (UID: "5b7c901a-2b97-4762-b1cf-3feaf6936dbc"). InnerVolumeSpecName "kube-api-access-z2tz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:46:20.427110 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.427053 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-token" (OuterVolumeSpecName: "istio-token") pod "5b7c901a-2b97-4762-b1cf-3feaf6936dbc" (UID: "5b7c901a-2b97-4762-b1cf-3feaf6936dbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:46:20.427110 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.427054 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5b7c901a-2b97-4762-b1cf-3feaf6936dbc" (UID: "5b7c901a-2b97-4762-b1cf-3feaf6936dbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 13:46:20.502902 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.502852 2580 generic.go:356] "Generic (PLEG): container finished" podID="5b7c901a-2b97-4762-b1cf-3feaf6936dbc" containerID="33c1e9ef8a20307bbb52acb785a5389fa998695d0a115eb0d2795aa867452f08" exitCode=0 Sep 16 13:46:20.503070 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.502913 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" event={"ID":"5b7c901a-2b97-4762-b1cf-3feaf6936dbc","Type":"ContainerDied","Data":"33c1e9ef8a20307bbb52acb785a5389fa998695d0a115eb0d2795aa867452f08"} Sep 16 13:46:20.503070 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.502929 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" Sep 16 13:46:20.503070 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.502970 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9" event={"ID":"5b7c901a-2b97-4762-b1cf-3feaf6936dbc","Type":"ContainerDied","Data":"a070253bd9768f6190819d089f9b10e1aa31a1f2850b99eeaba65ad718d30712"} Sep 16 13:46:20.503070 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.503000 2580 scope.go:117] "RemoveContainer" containerID="33c1e9ef8a20307bbb52acb785a5389fa998695d0a115eb0d2795aa867452f08" Sep 16 13:46:20.513306 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.513290 2580 scope.go:117] "RemoveContainer" containerID="92f1ab13a964bfb0d60a3f37be3c1c79ef0c2b159509f78d142217d968f7337d" Sep 16 13:46:20.521951 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.521933 2580 scope.go:117] "RemoveContainer" containerID="33c1e9ef8a20307bbb52acb785a5389fa998695d0a115eb0d2795aa867452f08" Sep 16 13:46:20.522243 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:46:20.522222 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c1e9ef8a20307bbb52acb785a5389fa998695d0a115eb0d2795aa867452f08\": container with ID starting with 33c1e9ef8a20307bbb52acb785a5389fa998695d0a115eb0d2795aa867452f08 not found: ID does not exist" containerID="33c1e9ef8a20307bbb52acb785a5389fa998695d0a115eb0d2795aa867452f08" Sep 16 13:46:20.522292 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.522253 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c1e9ef8a20307bbb52acb785a5389fa998695d0a115eb0d2795aa867452f08"} err="failed to get container status \"33c1e9ef8a20307bbb52acb785a5389fa998695d0a115eb0d2795aa867452f08\": rpc error: code = NotFound desc = could not find container \"33c1e9ef8a20307bbb52acb785a5389fa998695d0a115eb0d2795aa867452f08\": container with ID starting with 33c1e9ef8a20307bbb52acb785a5389fa998695d0a115eb0d2795aa867452f08 not found: ID does not exist" Sep 16 13:46:20.522292 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.522271 2580 scope.go:117] "RemoveContainer" containerID="92f1ab13a964bfb0d60a3f37be3c1c79ef0c2b159509f78d142217d968f7337d" Sep 16 13:46:20.522520 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:46:20.522503 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f1ab13a964bfb0d60a3f37be3c1c79ef0c2b159509f78d142217d968f7337d\": container with ID starting with 92f1ab13a964bfb0d60a3f37be3c1c79ef0c2b159509f78d142217d968f7337d not found: ID does not exist" containerID="92f1ab13a964bfb0d60a3f37be3c1c79ef0c2b159509f78d142217d968f7337d" Sep 16 13:46:20.522569 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.522528 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f1ab13a964bfb0d60a3f37be3c1c79ef0c2b159509f78d142217d968f7337d"} err="failed to get container status \"92f1ab13a964bfb0d60a3f37be3c1c79ef0c2b159509f78d142217d968f7337d\": rpc error: code = NotFound desc = could not find container \"92f1ab13a964bfb0d60a3f37be3c1c79ef0c2b159509f78d142217d968f7337d\": container with ID starting with 92f1ab13a964bfb0d60a3f37be3c1c79ef0c2b159509f78d142217d968f7337d not found: ID does not exist" Sep 16 13:46:20.525753 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.525730 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2tz7\" (UniqueName: \"kubernetes.io/projected/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-kube-api-access-z2tz7\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:46:20.525753 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.525754 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istiod-ca-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:46:20.525906 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.525764 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-podinfo\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:46:20.525906 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.525771 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-token\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:46:20.525906 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.525779 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7c901a-2b97-4762-b1cf-3feaf6936dbc-istio-envoy\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:46:20.527563 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.527544 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9"] Sep 16 13:46:20.532216 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.532193 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1a60997c-openshift-default-dcf575ccb-hjdw9"] Sep 16 13:46:20.571438 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:20.571404 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:21.453975 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:21.453944 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:22.355114 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:22.355083 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b7c901a-2b97-4762-b1cf-3feaf6936dbc" path="/var/lib/kubelet/pods/5b7c901a-2b97-4762-b1cf-3feaf6936dbc/volumes" Sep 16 13:46:22.415217 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:22.415174 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:23.204035 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:23.203988 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:24.112224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:24.112196 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:24.213133 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:24.213103 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:24.301203 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:24.301172 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:24.610242 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:24.610211 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:46:24.630866 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:24.630835 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:24.835545 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:24.835516 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:24.887792 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:24.887712 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk"] Sep 16 13:46:24.888105 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:24.888092 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0a207d3-89da-4c7b-9601-d14c289360b4" containerName="payload-pre-processing" Sep 16 13:46:24.888147 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:24.888111 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0a207d3-89da-4c7b-9601-d14c289360b4" containerName="payload-pre-processing" Sep 16 13:46:24.888147 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:24.888135 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b7c901a-2b97-4762-b1cf-3feaf6936dbc" containerName="istio-proxy" Sep 16 13:46:24.888147 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:24.888141 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b7c901a-2b97-4762-b1cf-3feaf6936dbc" containerName="istio-proxy" Sep 16 13:46:24.888249 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:24.888150 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b7c901a-2b97-4762-b1cf-3feaf6936dbc" containerName="istio-proxy" Sep 16 13:46:24.888249 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:24.888156 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b7c901a-2b97-4762-b1cf-3feaf6936dbc" containerName="istio-proxy" Sep 16 13:46:24.888249 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:24.888218 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0a207d3-89da-4c7b-9601-d14c289360b4" containerName="payload-pre-processing" Sep 16 13:46:24.888249 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:24.888226 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b7c901a-2b97-4762-b1cf-3feaf6936dbc" containerName="istio-proxy" Sep 16 13:46:24.888249 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:24.888233 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b7c901a-2b97-4762-b1cf-3feaf6936dbc" containerName="istio-proxy" Sep 16 13:46:24.892222 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:24.892203 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk" Sep 16 13:46:24.899725 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:24.899698 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-4985b031-dockercfg-bs6hs\"" Sep 16 13:46:24.899851 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:24.899707 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4985b031\"" Sep 16 13:46:24.906876 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:24.906852 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk"] Sep 16 13:46:24.965199 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:24.965161 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pndz\" (UniqueName: \"kubernetes.io/projected/0ff068bc-d68d-49a9-9429-a8a6f9d01e0e-kube-api-access-8pndz\") pod \"payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk\" (UID: \"0ff068bc-d68d-49a9-9429-a8a6f9d01e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk" Sep 16 13:46:24.965366 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:24.965278 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff068bc-d68d-49a9-9429-a8a6f9d01e0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk\" (UID: \"0ff068bc-d68d-49a9-9429-a8a6f9d01e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk" Sep 16 13:46:25.066514 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:25.066473 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff068bc-d68d-49a9-9429-a8a6f9d01e0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk\" (UID: \"0ff068bc-d68d-49a9-9429-a8a6f9d01e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk" Sep 16 13:46:25.066743 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:25.066538 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pndz\" (UniqueName: \"kubernetes.io/projected/0ff068bc-d68d-49a9-9429-a8a6f9d01e0e-kube-api-access-8pndz\") pod \"payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk\" (UID: \"0ff068bc-d68d-49a9-9429-a8a6f9d01e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk" Sep 16 13:46:25.067193 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:25.067171 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff068bc-d68d-49a9-9429-a8a6f9d01e0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk\" (UID: \"0ff068bc-d68d-49a9-9429-a8a6f9d01e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk" Sep 16 13:46:25.075595 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:25.075569 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pndz\" (UniqueName: \"kubernetes.io/projected/0ff068bc-d68d-49a9-9429-a8a6f9d01e0e-kube-api-access-8pndz\") pod \"payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk\" (UID: \"0ff068bc-d68d-49a9-9429-a8a6f9d01e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk" Sep 16 13:46:25.202819 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:25.202736 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk" Sep 16 13:46:25.241806 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:25.241772 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:25.477506 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:25.477450 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk"] Sep 16 13:46:25.479875 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:46:25.479835 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff068bc_d68d_49a9_9429_a8a6f9d01e0e.slice/crio-a33e05bf73f6233e8cbf8d36150c81d51c100a40b37bab3bd8fca9dcc5bf23c2 WatchSource:0}: Error finding container a33e05bf73f6233e8cbf8d36150c81d51c100a40b37bab3bd8fca9dcc5bf23c2: Status 404 returned error can't find the container with id a33e05bf73f6233e8cbf8d36150c81d51c100a40b37bab3bd8fca9dcc5bf23c2 Sep 16 13:46:25.532247 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:25.532215 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk" event={"ID":"0ff068bc-d68d-49a9-9429-a8a6f9d01e0e","Type":"ContainerStarted","Data":"a33e05bf73f6233e8cbf8d36150c81d51c100a40b37bab3bd8fca9dcc5bf23c2"} Sep 16 13:46:26.188148 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:26.188112 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:26.540023 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:26.539988 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk" event={"ID":"0ff068bc-d68d-49a9-9429-a8a6f9d01e0e","Type":"ContainerStarted","Data":"98290b48f73c84faf0ed0a6875ef988b132f012f7c5f4554052b1d4fa1b564a6"} Sep 16 13:46:26.540415 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:26.540122 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk" Sep 16 13:46:26.589770 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:26.589719 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk" podStartSLOduration=2.589699726 podStartE2EDuration="2.589699726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:46:26.586856195 +0000 UTC m=+934.687668909" watchObservedRunningTime="2026-09-16 13:46:26.589699726 +0000 UTC m=+934.690512441" Sep 16 13:46:27.309587 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:27.309557 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:28.054769 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:28.054738 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:28.825809 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:28.825772 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:29.637330 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:29.637299 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:30.539742 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:30.539706 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:33.769872 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:33.769836 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t"] Sep 16 13:46:33.794761 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:33.794691 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:34.449031 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:34.449000 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:35.209447 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.209413 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:35.219403 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.219378 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-14f2a6-29826105-vjstl"] Sep 16 13:46:35.225158 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.225123 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-14f2a6-29826105-vjstl"] Sep 16 13:46:35.256241 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.256205 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98"] Sep 16 13:46:35.256459 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.256431 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98" podUID="a413c643-7e13-4e01-9e54-5f6aab83e565" containerName="payload-processing" containerID="cri-o://a8a89269f10f552ae36e4ef6d6e5ad73c6f20d4f43b5ae33024085be75b54095" gracePeriod=30 Sep 16 13:46:35.511293 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.511266 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98" Sep 16 13:46:35.571983 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.571946 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmckj\" (UniqueName: \"kubernetes.io/projected/a413c643-7e13-4e01-9e54-5f6aab83e565-kube-api-access-lmckj\") pod \"a413c643-7e13-4e01-9e54-5f6aab83e565\" (UID: \"a413c643-7e13-4e01-9e54-5f6aab83e565\") " Sep 16 13:46:35.572142 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.572032 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a413c643-7e13-4e01-9e54-5f6aab83e565-plugins-config-volume\") pod \"a413c643-7e13-4e01-9e54-5f6aab83e565\" (UID: \"a413c643-7e13-4e01-9e54-5f6aab83e565\") " Sep 16 13:46:35.572389 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.572359 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a413c643-7e13-4e01-9e54-5f6aab83e565-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a413c643-7e13-4e01-9e54-5f6aab83e565" (UID: "a413c643-7e13-4e01-9e54-5f6aab83e565"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:46:35.572512 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.572456 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a413c643-7e13-4e01-9e54-5f6aab83e565-plugins-config-volume\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:46:35.574272 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.574247 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a413c643-7e13-4e01-9e54-5f6aab83e565-kube-api-access-lmckj" (OuterVolumeSpecName: "kube-api-access-lmckj") pod "a413c643-7e13-4e01-9e54-5f6aab83e565" (UID: "a413c643-7e13-4e01-9e54-5f6aab83e565"). InnerVolumeSpecName "kube-api-access-lmckj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:46:35.587792 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.587763 2580 generic.go:356] "Generic (PLEG): container finished" podID="a413c643-7e13-4e01-9e54-5f6aab83e565" containerID="a8a89269f10f552ae36e4ef6d6e5ad73c6f20d4f43b5ae33024085be75b54095" exitCode=0 Sep 16 13:46:35.587937 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.587809 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98" Sep 16 13:46:35.587937 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.587844 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98" event={"ID":"a413c643-7e13-4e01-9e54-5f6aab83e565","Type":"ContainerDied","Data":"a8a89269f10f552ae36e4ef6d6e5ad73c6f20d4f43b5ae33024085be75b54095"} Sep 16 13:46:35.587937 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.587903 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98" event={"ID":"a413c643-7e13-4e01-9e54-5f6aab83e565","Type":"ContainerDied","Data":"1253b362c2c98495a430bd53abfd64e64d8b2f89af16b34c13262bb033c71559"} Sep 16 13:46:35.587937 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.587932 2580 scope.go:117] "RemoveContainer" containerID="a8a89269f10f552ae36e4ef6d6e5ad73c6f20d4f43b5ae33024085be75b54095" Sep 16 13:46:35.597061 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.597045 2580 scope.go:117] "RemoveContainer" containerID="a8a89269f10f552ae36e4ef6d6e5ad73c6f20d4f43b5ae33024085be75b54095" Sep 16 13:46:35.597322 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:46:35.597302 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8a89269f10f552ae36e4ef6d6e5ad73c6f20d4f43b5ae33024085be75b54095\": container with ID starting with a8a89269f10f552ae36e4ef6d6e5ad73c6f20d4f43b5ae33024085be75b54095 not found: ID does not exist" containerID="a8a89269f10f552ae36e4ef6d6e5ad73c6f20d4f43b5ae33024085be75b54095" Sep 16 13:46:35.597388 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.597328 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a89269f10f552ae36e4ef6d6e5ad73c6f20d4f43b5ae33024085be75b54095"} err="failed to get container status \"a8a89269f10f552ae36e4ef6d6e5ad73c6f20d4f43b5ae33024085be75b54095\": rpc error: code = NotFound desc = could not find container \"a8a89269f10f552ae36e4ef6d6e5ad73c6f20d4f43b5ae33024085be75b54095\": container with ID starting with a8a89269f10f552ae36e4ef6d6e5ad73c6f20d4f43b5ae33024085be75b54095 not found: ID does not exist" Sep 16 13:46:35.610746 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.610718 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98"] Sep 16 13:46:35.616923 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.616905 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-14f2a6-85c8d4d7bf-f8j98"] Sep 16 13:46:35.673868 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.673839 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmckj\" (UniqueName: \"kubernetes.io/projected/a413c643-7e13-4e01-9e54-5f6aab83e565-kube-api-access-lmckj\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:46:35.730776 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.730710 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:35.935195 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:35.935097 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:36.055983 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:36.055954 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:36.354991 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:36.354959 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a824cc3-306c-426c-ae15-e6b8a742bab0" path="/var/lib/kubelet/pods/1a824cc3-306c-426c-ae15-e6b8a742bab0/volumes" Sep 16 13:46:36.355374 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:36.355289 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a413c643-7e13-4e01-9e54-5f6aab83e565" path="/var/lib/kubelet/pods/a413c643-7e13-4e01-9e54-5f6aab83e565/volumes" Sep 16 13:46:37.163854 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:37.163826 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:37.547733 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:37.547698 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk" Sep 16 13:46:38.281075 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:38.281044 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:39.030257 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:39.030224 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:39.878960 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:39.878930 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:40.650979 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:40.650945 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:40.990737 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:40.990701 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:41.648213 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:41.648175 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:42.561756 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:42.561720 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:43.284742 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:43.284704 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:44.261236 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:44.261205 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:45.307202 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:45.307161 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:46.021525 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:46.021495 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:46.728653 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:46.728621 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:47.623705 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:47.623673 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:48.452274 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:48.452243 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:48.789343 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:48.789299 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" podUID="1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee" containerName="main" containerID="cri-o://07a69cb29b0b2e208e465d0b816afff15770924aecfcf7321ca3f45f4fc643d5" gracePeriod=45 Sep 16 13:46:49.036827 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.036800 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:46:49.101583 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.101496 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-model-cache\") pod \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " Sep 16 13:46:49.101583 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.101582 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-dshm\") pod \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " Sep 16 13:46:49.101801 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.101637 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-tls-certs\") pod \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " Sep 16 13:46:49.101801 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.101674 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-tmp-dir\") pod \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " Sep 16 13:46:49.101801 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.101698 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-home\") pod \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " Sep 16 13:46:49.101801 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.101724 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kz6pw\" (UniqueName: \"kubernetes.io/projected/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-kube-api-access-kz6pw\") pod \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\" (UID: \"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee\") " Sep 16 13:46:49.102000 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.101804 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-model-cache" (OuterVolumeSpecName: "model-cache") pod "1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee" (UID: "1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:46:49.102000 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.101951 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-home" (OuterVolumeSpecName: "home") pod "1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee" (UID: "1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:46:49.102000 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.101960 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee" (UID: "1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:46:49.102155 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.102129 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:46:49.102155 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.102148 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:46:49.102239 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.102162 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:46:49.104061 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.104030 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee" (UID: "1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:46:49.104061 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.104046 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-dshm" (OuterVolumeSpecName: "dshm") pod "1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee" (UID: "1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:46:49.104210 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.104063 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-kube-api-access-kz6pw" (OuterVolumeSpecName: "kube-api-access-kz6pw") pod "1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee" (UID: "1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee"). InnerVolumeSpecName "kube-api-access-kz6pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:46:49.203028 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.202993 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:46:49.203028 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.203020 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:46:49.203028 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.203030 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz6pw\" (UniqueName: \"kubernetes.io/projected/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee-kube-api-access-kz6pw\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:46:49.666028 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.665998 2580 generic.go:356] "Generic (PLEG): container finished" podID="1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee" containerID="07a69cb29b0b2e208e465d0b816afff15770924aecfcf7321ca3f45f4fc643d5" exitCode=0 Sep 16 13:46:49.666444 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.666043 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" Sep 16 13:46:49.666444 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.666071 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" event={"ID":"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee","Type":"ContainerDied","Data":"07a69cb29b0b2e208e465d0b816afff15770924aecfcf7321ca3f45f4fc643d5"} Sep 16 13:46:49.666444 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.666113 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t" event={"ID":"1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee","Type":"ContainerDied","Data":"238e35240271caa6a35eb91ccb912145a8e7c23ede69fa0e362018b0de059967"} Sep 16 13:46:49.666444 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.666131 2580 scope.go:117] "RemoveContainer" containerID="07a69cb29b0b2e208e465d0b816afff15770924aecfcf7321ca3f45f4fc643d5" Sep 16 13:46:49.675381 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.675363 2580 scope.go:117] "RemoveContainer" containerID="07a69cb29b0b2e208e465d0b816afff15770924aecfcf7321ca3f45f4fc643d5" Sep 16 13:46:49.675675 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:46:49.675653 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07a69cb29b0b2e208e465d0b816afff15770924aecfcf7321ca3f45f4fc643d5\": container with ID starting with 07a69cb29b0b2e208e465d0b816afff15770924aecfcf7321ca3f45f4fc643d5 not found: ID does not exist" containerID="07a69cb29b0b2e208e465d0b816afff15770924aecfcf7321ca3f45f4fc643d5" Sep 16 13:46:49.675742 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.675685 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07a69cb29b0b2e208e465d0b816afff15770924aecfcf7321ca3f45f4fc643d5"} err="failed to get container status \"07a69cb29b0b2e208e465d0b816afff15770924aecfcf7321ca3f45f4fc643d5\": rpc error: code = NotFound desc = could not find container \"07a69cb29b0b2e208e465d0b816afff15770924aecfcf7321ca3f45f4fc643d5\": container with ID starting with 07a69cb29b0b2e208e465d0b816afff15770924aecfcf7321ca3f45f4fc643d5 not found: ID does not exist" Sep 16 13:46:49.688924 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.688898 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t"] Sep 16 13:46:49.695428 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:49.695405 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-14f2a6/autores-model-14f2a6-kserve-84c5b797dc-8g59t"] Sep 16 13:46:50.330170 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:50.330140 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:50.355253 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:50.355221 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee" path="/var/lib/kubelet/pods/1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee/volumes" Sep 16 13:46:50.940894 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:50.940857 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:50.972688 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:50.972659 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk"] Sep 16 13:46:50.972916 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:50.972892 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk" podUID="0ff068bc-d68d-49a9-9429-a8a6f9d01e0e" containerName="payload-pre-processing" containerID="cri-o://98290b48f73c84faf0ed0a6875ef988b132f012f7c5f4554052b1d4fa1b564a6" gracePeriod=30 Sep 16 13:46:51.042359 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.042316 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:51.142422 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.142385 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:51.221983 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.221957 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk" Sep 16 13:46:51.240374 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.240346 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:51.321565 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.321533 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff068bc-d68d-49a9-9429-a8a6f9d01e0e-plugins-config-volume\") pod \"0ff068bc-d68d-49a9-9429-a8a6f9d01e0e\" (UID: \"0ff068bc-d68d-49a9-9429-a8a6f9d01e0e\") " Sep 16 13:46:51.321796 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.321639 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pndz\" (UniqueName: \"kubernetes.io/projected/0ff068bc-d68d-49a9-9429-a8a6f9d01e0e-kube-api-access-8pndz\") pod \"0ff068bc-d68d-49a9-9429-a8a6f9d01e0e\" (UID: \"0ff068bc-d68d-49a9-9429-a8a6f9d01e0e\") " Sep 16 13:46:51.321907 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.321884 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff068bc-d68d-49a9-9429-a8a6f9d01e0e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0ff068bc-d68d-49a9-9429-a8a6f9d01e0e" (UID: "0ff068bc-d68d-49a9-9429-a8a6f9d01e0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:46:51.321994 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.321980 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff068bc-d68d-49a9-9429-a8a6f9d01e0e-plugins-config-volume\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:46:51.324910 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.324872 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff068bc-d68d-49a9-9429-a8a6f9d01e0e-kube-api-access-8pndz" (OuterVolumeSpecName: "kube-api-access-8pndz") pod "0ff068bc-d68d-49a9-9429-a8a6f9d01e0e" (UID: "0ff068bc-d68d-49a9-9429-a8a6f9d01e0e"). InnerVolumeSpecName "kube-api-access-8pndz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:46:51.412147 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.412114 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:51.423382 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.423357 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pndz\" (UniqueName: \"kubernetes.io/projected/0ff068bc-d68d-49a9-9429-a8a6f9d01e0e-kube-api-access-8pndz\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:46:51.635418 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.635384 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:51.666863 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.666826 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g"] Sep 16 13:46:51.667173 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.667160 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a413c643-7e13-4e01-9e54-5f6aab83e565" containerName="payload-processing" Sep 16 13:46:51.667231 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.667175 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a413c643-7e13-4e01-9e54-5f6aab83e565" containerName="payload-processing" Sep 16 13:46:51.667231 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.667192 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee" containerName="main" Sep 16 13:46:51.667231 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.667197 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee" containerName="main" Sep 16 13:46:51.667231 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.667204 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ff068bc-d68d-49a9-9429-a8a6f9d01e0e" containerName="payload-pre-processing" Sep 16 13:46:51.667231 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.667209 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ff068bc-d68d-49a9-9429-a8a6f9d01e0e" containerName="payload-pre-processing" Sep 16 13:46:51.667443 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.667307 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fc4f13d-201c-46ec-bf3a-be2c2b54b6ee" containerName="main" Sep 16 13:46:51.667443 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.667316 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ff068bc-d68d-49a9-9429-a8a6f9d01e0e" containerName="payload-pre-processing" Sep 16 13:46:51.667443 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.667323 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a413c643-7e13-4e01-9e54-5f6aab83e565" containerName="payload-processing" Sep 16 13:46:51.670639 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.670596 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:46:51.673687 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.673668 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-46ccbd\"/\"premium-w0-46ccbd-kserve-self-signed-certs\"" Sep 16 13:46:51.682119 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.682091 2580 generic.go:356] "Generic (PLEG): container finished" podID="0ff068bc-d68d-49a9-9429-a8a6f9d01e0e" containerID="98290b48f73c84faf0ed0a6875ef988b132f012f7c5f4554052b1d4fa1b564a6" exitCode=0 Sep 16 13:46:51.682232 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.682130 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk" Sep 16 13:46:51.682232 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.682175 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk" event={"ID":"0ff068bc-d68d-49a9-9429-a8a6f9d01e0e","Type":"ContainerDied","Data":"98290b48f73c84faf0ed0a6875ef988b132f012f7c5f4554052b1d4fa1b564a6"} Sep 16 13:46:51.682309 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.682236 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk" event={"ID":"0ff068bc-d68d-49a9-9429-a8a6f9d01e0e","Type":"ContainerDied","Data":"a33e05bf73f6233e8cbf8d36150c81d51c100a40b37bab3bd8fca9dcc5bf23c2"} Sep 16 13:46:51.682309 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.682266 2580 scope.go:117] "RemoveContainer" containerID="98290b48f73c84faf0ed0a6875ef988b132f012f7c5f4554052b1d4fa1b564a6" Sep 16 13:46:51.682625 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.682583 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g"] Sep 16 13:46:51.693356 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.693318 2580 scope.go:117] "RemoveContainer" containerID="98290b48f73c84faf0ed0a6875ef988b132f012f7c5f4554052b1d4fa1b564a6" Sep 16 13:46:51.693838 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:46:51.693814 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98290b48f73c84faf0ed0a6875ef988b132f012f7c5f4554052b1d4fa1b564a6\": container with ID starting with 98290b48f73c84faf0ed0a6875ef988b132f012f7c5f4554052b1d4fa1b564a6 not found: ID does not exist" containerID="98290b48f73c84faf0ed0a6875ef988b132f012f7c5f4554052b1d4fa1b564a6" Sep 16 13:46:51.693928 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.693847 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98290b48f73c84faf0ed0a6875ef988b132f012f7c5f4554052b1d4fa1b564a6"} err="failed to get container status \"98290b48f73c84faf0ed0a6875ef988b132f012f7c5f4554052b1d4fa1b564a6\": rpc error: code = NotFound desc = could not find container \"98290b48f73c84faf0ed0a6875ef988b132f012f7c5f4554052b1d4fa1b564a6\": container with ID starting with 98290b48f73c84faf0ed0a6875ef988b132f012f7c5f4554052b1d4fa1b564a6 not found: ID does not exist" Sep 16 13:46:51.713899 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.713875 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk"] Sep 16 13:46:51.721518 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.721496 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4985b031-69fd474d86-l7glk"] Sep 16 13:46:51.726191 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.726176 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-model-cache\") pod \"premium-w0-46ccbd-kserve-67784f9874-92p9g\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:46:51.726246 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.726205 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-dshm\") pod \"premium-w0-46ccbd-kserve-67784f9874-92p9g\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:46:51.726283 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.726274 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qphxl\" (UniqueName: \"kubernetes.io/projected/4f129b28-ba14-4640-b632-b9eb00e9485e-kube-api-access-qphxl\") pod \"premium-w0-46ccbd-kserve-67784f9874-92p9g\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:46:51.726330 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.726318 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-tmp-dir\") pod \"premium-w0-46ccbd-kserve-67784f9874-92p9g\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:46:51.726368 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.726338 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f129b28-ba14-4640-b632-b9eb00e9485e-tls-certs\") pod \"premium-w0-46ccbd-kserve-67784f9874-92p9g\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:46:51.726404 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.726383 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-home\") pod \"premium-w0-46ccbd-kserve-67784f9874-92p9g\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:46:51.827407 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.827365 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-tmp-dir\") pod \"premium-w0-46ccbd-kserve-67784f9874-92p9g\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:46:51.827407 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.827406 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f129b28-ba14-4640-b632-b9eb00e9485e-tls-certs\") pod \"premium-w0-46ccbd-kserve-67784f9874-92p9g\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:46:51.827692 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.827519 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-home\") pod \"premium-w0-46ccbd-kserve-67784f9874-92p9g\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:46:51.827757 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.827725 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-model-cache\") pod \"premium-w0-46ccbd-kserve-67784f9874-92p9g\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:46:51.827839 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.827819 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-dshm\") pod \"premium-w0-46ccbd-kserve-67784f9874-92p9g\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:46:51.827922 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.827866 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-tmp-dir\") pod \"premium-w0-46ccbd-kserve-67784f9874-92p9g\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:46:51.827986 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.827961 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qphxl\" (UniqueName: \"kubernetes.io/projected/4f129b28-ba14-4640-b632-b9eb00e9485e-kube-api-access-qphxl\") pod \"premium-w0-46ccbd-kserve-67784f9874-92p9g\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:46:51.828043 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.827999 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-model-cache\") pod \"premium-w0-46ccbd-kserve-67784f9874-92p9g\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:46:51.828105 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.828085 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-home\") pod \"premium-w0-46ccbd-kserve-67784f9874-92p9g\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:46:51.830448 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.830417 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-dshm\") pod \"premium-w0-46ccbd-kserve-67784f9874-92p9g\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:46:51.830697 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.830679 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f129b28-ba14-4640-b632-b9eb00e9485e-tls-certs\") pod \"premium-w0-46ccbd-kserve-67784f9874-92p9g\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:46:51.836045 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.836021 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qphxl\" (UniqueName: \"kubernetes.io/projected/4f129b28-ba14-4640-b632-b9eb00e9485e-kube-api-access-qphxl\") pod \"premium-w0-46ccbd-kserve-67784f9874-92p9g\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:46:51.981484 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:51.981454 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:46:52.085889 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:52.085853 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:52.184710 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:52.184676 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:52.264930 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:52.264906 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g"] Sep 16 13:46:52.267733 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:46:52.267686 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f129b28_ba14_4640_b632_b9eb00e9485e.slice/crio-41e2b3a17c49f568c0c95da384983c22fd041051af6c3e12d543f7b15268f737 WatchSource:0}: Error finding container 41e2b3a17c49f568c0c95da384983c22fd041051af6c3e12d543f7b15268f737: Status 404 returned error can't find the container with id 41e2b3a17c49f568c0c95da384983c22fd041051af6c3e12d543f7b15268f737 Sep 16 13:46:52.268306 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:52.268253 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:52.355669 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:52.355636 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ff068bc-d68d-49a9-9429-a8a6f9d01e0e" path="/var/lib/kubelet/pods/0ff068bc-d68d-49a9-9429-a8a6f9d01e0e/volumes" Sep 16 13:46:52.521748 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:52.521657 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:52.689111 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:52.689080 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" event={"ID":"4f129b28-ba14-4640-b632-b9eb00e9485e","Type":"ContainerStarted","Data":"1b231c1ffaaf37b02b6df54bbdc686e94c3871513796aca21511bc8418258cfc"} Sep 16 13:46:52.689111 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:52.689116 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" event={"ID":"4f129b28-ba14-4640-b632-b9eb00e9485e","Type":"ContainerStarted","Data":"41e2b3a17c49f568c0c95da384983c22fd041051af6c3e12d543f7b15268f737"} Sep 16 13:46:52.710881 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:52.710829 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" podStartSLOduration=1.710815845 podStartE2EDuration="1.710815845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:46:52.709894137 +0000 UTC m=+960.810706854" watchObservedRunningTime="2026-09-16 13:46:52.710815845 +0000 UTC m=+960.811628613" Sep 16 13:46:53.240854 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:53.240815 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:46:53.416953 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:53.416917 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:53.779876 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:53.779841 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:54.214735 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:54.214709 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:54.876929 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:46:54.876896 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:00.291818 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:00.291782 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:00.410370 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:00.410335 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:00.499738 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:00.499700 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:00.628138 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:00.628025 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:01.007270 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:01.007236 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:01.104088 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:01.104057 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:01.326447 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:01.326364 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t"] Sep 16 13:47:01.326836 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:01.326571 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" podUID="582d3a17-df42-4551-b213-9fc1a00a77d9" containerName="istio-proxy" containerID="cri-o://1bcdaf348d5c9578ad70f52878963ce1c3eb9c545cf55fcd7ce89b2da3641351" gracePeriod=30 Sep 16 13:47:01.398591 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:01.398554 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:01.758137 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:01.758105 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:01.982056 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:01.982023 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:47:01.982235 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:01.982080 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:47:01.995525 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:01.995498 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:47:02.065548 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:02.065481 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:02.195289 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:02.195253 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:02.751279 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:02.751251 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:47:05.050235 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:05.050198 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:05.681141 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:05.681105 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:05.759692 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:05.759657 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:05.861445 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:05.861412 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:06.578456 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.578433 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:47:06.673389 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.673291 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-data\") pod \"582d3a17-df42-4551-b213-9fc1a00a77d9\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " Sep 16 13:47:06.673389 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.673341 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-podinfo\") pod \"582d3a17-df42-4551-b213-9fc1a00a77d9\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " Sep 16 13:47:06.673389 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.673380 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-workload-certs\") pod \"582d3a17-df42-4551-b213-9fc1a00a77d9\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " Sep 16 13:47:06.673703 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.673428 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-workload-socket\") pod \"582d3a17-df42-4551-b213-9fc1a00a77d9\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " Sep 16 13:47:06.673703 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.673457 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-envoy\") pod \"582d3a17-df42-4551-b213-9fc1a00a77d9\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " Sep 16 13:47:06.673703 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.673492 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-token\") pod \"582d3a17-df42-4551-b213-9fc1a00a77d9\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " Sep 16 13:47:06.673703 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.673530 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpbm8\" (UniqueName: \"kubernetes.io/projected/582d3a17-df42-4551-b213-9fc1a00a77d9-kube-api-access-xpbm8\") pod \"582d3a17-df42-4551-b213-9fc1a00a77d9\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " Sep 16 13:47:06.673703 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.673556 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/582d3a17-df42-4551-b213-9fc1a00a77d9-istiod-ca-cert\") pod \"582d3a17-df42-4551-b213-9fc1a00a77d9\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " Sep 16 13:47:06.673703 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.673588 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-credential-socket\") pod \"582d3a17-df42-4551-b213-9fc1a00a77d9\" (UID: \"582d3a17-df42-4551-b213-9fc1a00a77d9\") " Sep 16 13:47:06.673703 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.673634 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-data" (OuterVolumeSpecName: "istio-data") pod "582d3a17-df42-4551-b213-9fc1a00a77d9" (UID: "582d3a17-df42-4551-b213-9fc1a00a77d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:47:06.674052 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.673709 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "582d3a17-df42-4551-b213-9fc1a00a77d9" (UID: "582d3a17-df42-4551-b213-9fc1a00a77d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:47:06.674052 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.673782 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "582d3a17-df42-4551-b213-9fc1a00a77d9" (UID: "582d3a17-df42-4551-b213-9fc1a00a77d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:47:06.674052 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.673949 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "582d3a17-df42-4551-b213-9fc1a00a77d9" (UID: "582d3a17-df42-4551-b213-9fc1a00a77d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:47:06.674052 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.674015 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/582d3a17-df42-4551-b213-9fc1a00a77d9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "582d3a17-df42-4551-b213-9fc1a00a77d9" (UID: "582d3a17-df42-4551-b213-9fc1a00a77d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:47:06.674052 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.674025 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-data\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:47:06.674249 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.674067 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-workload-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:47:06.674249 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.674084 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-workload-socket\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:47:06.676063 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.676030 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582d3a17-df42-4551-b213-9fc1a00a77d9-kube-api-access-xpbm8" (OuterVolumeSpecName: "kube-api-access-xpbm8") pod "582d3a17-df42-4551-b213-9fc1a00a77d9" (UID: "582d3a17-df42-4551-b213-9fc1a00a77d9"). InnerVolumeSpecName "kube-api-access-xpbm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:47:06.676176 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.676086 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-token" (OuterVolumeSpecName: "istio-token") pod "582d3a17-df42-4551-b213-9fc1a00a77d9" (UID: "582d3a17-df42-4551-b213-9fc1a00a77d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:47:06.676332 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.676310 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "582d3a17-df42-4551-b213-9fc1a00a77d9" (UID: "582d3a17-df42-4551-b213-9fc1a00a77d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 13:47:06.676332 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.676318 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "582d3a17-df42-4551-b213-9fc1a00a77d9" (UID: "582d3a17-df42-4551-b213-9fc1a00a77d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:47:06.761843 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.761811 2580 generic.go:356] "Generic (PLEG): container finished" podID="582d3a17-df42-4551-b213-9fc1a00a77d9" containerID="1bcdaf348d5c9578ad70f52878963ce1c3eb9c545cf55fcd7ce89b2da3641351" exitCode=0 Sep 16 13:47:06.761990 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.761862 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" Sep 16 13:47:06.761990 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.761869 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" event={"ID":"582d3a17-df42-4551-b213-9fc1a00a77d9","Type":"ContainerDied","Data":"1bcdaf348d5c9578ad70f52878963ce1c3eb9c545cf55fcd7ce89b2da3641351"} Sep 16 13:47:06.761990 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.761912 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t" event={"ID":"582d3a17-df42-4551-b213-9fc1a00a77d9","Type":"ContainerDied","Data":"7783ced512ec1c95c3e44eba4dd912cf3a706a3d34f52a974973095d69f00a71"} Sep 16 13:47:06.761990 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.761934 2580 scope.go:117] "RemoveContainer" containerID="1bcdaf348d5c9578ad70f52878963ce1c3eb9c545cf55fcd7ce89b2da3641351" Sep 16 13:47:06.771505 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.771484 2580 scope.go:117] "RemoveContainer" containerID="76bdcd11c690c4f5af4657e4d823983edccc51dd4ede8ffd64f903673ea37e81" Sep 16 13:47:06.774859 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.774834 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-podinfo\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:47:06.774859 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.774857 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-envoy\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:47:06.774998 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.774865 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/582d3a17-df42-4551-b213-9fc1a00a77d9-istio-token\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:47:06.774998 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.774873 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpbm8\" (UniqueName: \"kubernetes.io/projected/582d3a17-df42-4551-b213-9fc1a00a77d9-kube-api-access-xpbm8\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:47:06.774998 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.774882 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/582d3a17-df42-4551-b213-9fc1a00a77d9-istiod-ca-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:47:06.774998 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.774892 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/582d3a17-df42-4551-b213-9fc1a00a77d9-credential-socket\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:47:06.780063 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.780046 2580 scope.go:117] "RemoveContainer" containerID="1bcdaf348d5c9578ad70f52878963ce1c3eb9c545cf55fcd7ce89b2da3641351" Sep 16 13:47:06.780345 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:47:06.780324 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bcdaf348d5c9578ad70f52878963ce1c3eb9c545cf55fcd7ce89b2da3641351\": container with ID starting with 1bcdaf348d5c9578ad70f52878963ce1c3eb9c545cf55fcd7ce89b2da3641351 not found: ID does not exist" containerID="1bcdaf348d5c9578ad70f52878963ce1c3eb9c545cf55fcd7ce89b2da3641351" Sep 16 13:47:06.780390 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.780355 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bcdaf348d5c9578ad70f52878963ce1c3eb9c545cf55fcd7ce89b2da3641351"} err="failed to get container status \"1bcdaf348d5c9578ad70f52878963ce1c3eb9c545cf55fcd7ce89b2da3641351\": rpc error: code = NotFound desc = could not find container \"1bcdaf348d5c9578ad70f52878963ce1c3eb9c545cf55fcd7ce89b2da3641351\": container with ID starting with 1bcdaf348d5c9578ad70f52878963ce1c3eb9c545cf55fcd7ce89b2da3641351 not found: ID does not exist" Sep 16 13:47:06.780390 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.780372 2580 scope.go:117] "RemoveContainer" containerID="76bdcd11c690c4f5af4657e4d823983edccc51dd4ede8ffd64f903673ea37e81" Sep 16 13:47:06.780633 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:47:06.780598 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76bdcd11c690c4f5af4657e4d823983edccc51dd4ede8ffd64f903673ea37e81\": container with ID starting with 76bdcd11c690c4f5af4657e4d823983edccc51dd4ede8ffd64f903673ea37e81 not found: ID does not exist" containerID="76bdcd11c690c4f5af4657e4d823983edccc51dd4ede8ffd64f903673ea37e81" Sep 16 13:47:06.780682 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.780639 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76bdcd11c690c4f5af4657e4d823983edccc51dd4ede8ffd64f903673ea37e81"} err="failed to get container status \"76bdcd11c690c4f5af4657e4d823983edccc51dd4ede8ffd64f903673ea37e81\": rpc error: code = NotFound desc = could not find container \"76bdcd11c690c4f5af4657e4d823983edccc51dd4ede8ffd64f903673ea37e81\": container with ID starting with 76bdcd11c690c4f5af4657e4d823983edccc51dd4ede8ffd64f903673ea37e81 not found: ID does not exist" Sep 16 13:47:06.785412 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.785389 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t"] Sep 16 13:47:06.791746 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.791723 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-14f2a6-openshift-default-55c76864b-gdt7t"] Sep 16 13:47:06.922128 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:06.922090 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:07.644213 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:07.644180 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:08.217084 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:08.217045 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:47:08.355890 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:08.355858 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="582d3a17-df42-4551-b213-9fc1a00a77d9" path="/var/lib/kubelet/pods/582d3a17-df42-4551-b213-9fc1a00a77d9/volumes" Sep 16 13:47:08.377408 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:08.377380 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:09.251825 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:09.251778 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:09.316573 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:09.316545 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:09.699741 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:09.699655 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:09.803706 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:09.803666 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:09.812846 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:09.812814 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj"] Sep 16 13:47:09.813188 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:09.813171 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="582d3a17-df42-4551-b213-9fc1a00a77d9" containerName="istio-proxy" Sep 16 13:47:09.813188 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:09.813187 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="582d3a17-df42-4551-b213-9fc1a00a77d9" containerName="istio-proxy" Sep 16 13:47:09.813325 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:09.813276 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="582d3a17-df42-4551-b213-9fc1a00a77d9" containerName="istio-proxy" Sep 16 13:47:09.813325 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:09.813292 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="582d3a17-df42-4551-b213-9fc1a00a77d9" containerName="istio-proxy" Sep 16 13:47:09.817579 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:09.817563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj" Sep 16 13:47:09.820178 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:09.820162 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-4678b2-dockercfg-nnrpt\"" Sep 16 13:47:09.820591 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:09.820574 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-4678b2\"" Sep 16 13:47:09.825923 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:09.825902 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj"] Sep 16 13:47:09.837676 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:09.837653 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt"] Sep 16 13:47:09.838045 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:09.838028 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="582d3a17-df42-4551-b213-9fc1a00a77d9" containerName="istio-proxy" Sep 16 13:47:09.838125 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:09.838049 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="582d3a17-df42-4551-b213-9fc1a00a77d9" containerName="istio-proxy" Sep 16 13:47:09.846961 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:09.846934 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt" Sep 16 13:47:09.851058 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:09.851031 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt"] Sep 16 13:47:10.005363 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.005324 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79rv5\" (UniqueName: \"kubernetes.io/projected/db402416-c8de-4346-83fb-5997122a7272-kube-api-access-79rv5\") pod \"payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt\" (UID: \"db402416-c8de-4346-83fb-5997122a7272\") " pod="openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt" Sep 16 13:47:10.005546 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.005399 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db402416-c8de-4346-83fb-5997122a7272-plugins-config-volume\") pod \"payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt\" (UID: \"db402416-c8de-4346-83fb-5997122a7272\") " pod="openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt" Sep 16 13:47:10.005546 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.005484 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f45sf\" (UniqueName: \"kubernetes.io/projected/1d67a8d4-e44a-4604-b653-fa7cd1b6d90e-kube-api-access-f45sf\") pod \"payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj\" (UID: \"1d67a8d4-e44a-4604-b653-fa7cd1b6d90e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj" Sep 16 13:47:10.005546 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.005518 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d67a8d4-e44a-4604-b653-fa7cd1b6d90e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj\" (UID: \"1d67a8d4-e44a-4604-b653-fa7cd1b6d90e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj" Sep 16 13:47:10.106533 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.106495 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79rv5\" (UniqueName: \"kubernetes.io/projected/db402416-c8de-4346-83fb-5997122a7272-kube-api-access-79rv5\") pod \"payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt\" (UID: \"db402416-c8de-4346-83fb-5997122a7272\") " pod="openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt" Sep 16 13:47:10.106774 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.106563 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db402416-c8de-4346-83fb-5997122a7272-plugins-config-volume\") pod \"payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt\" (UID: \"db402416-c8de-4346-83fb-5997122a7272\") " pod="openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt" Sep 16 13:47:10.106774 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.106596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f45sf\" (UniqueName: \"kubernetes.io/projected/1d67a8d4-e44a-4604-b653-fa7cd1b6d90e-kube-api-access-f45sf\") pod \"payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj\" (UID: \"1d67a8d4-e44a-4604-b653-fa7cd1b6d90e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj" Sep 16 13:47:10.106774 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.106638 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d67a8d4-e44a-4604-b653-fa7cd1b6d90e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj\" (UID: \"1d67a8d4-e44a-4604-b653-fa7cd1b6d90e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj" Sep 16 13:47:10.107309 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.107287 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db402416-c8de-4346-83fb-5997122a7272-plugins-config-volume\") pod \"payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt\" (UID: \"db402416-c8de-4346-83fb-5997122a7272\") " pod="openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt" Sep 16 13:47:10.107348 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.107315 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d67a8d4-e44a-4604-b653-fa7cd1b6d90e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj\" (UID: \"1d67a8d4-e44a-4604-b653-fa7cd1b6d90e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj" Sep 16 13:47:10.114832 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.114804 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79rv5\" (UniqueName: \"kubernetes.io/projected/db402416-c8de-4346-83fb-5997122a7272-kube-api-access-79rv5\") pod \"payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt\" (UID: \"db402416-c8de-4346-83fb-5997122a7272\") " pod="openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt" Sep 16 13:47:10.114977 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.114951 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f45sf\" (UniqueName: \"kubernetes.io/projected/1d67a8d4-e44a-4604-b653-fa7cd1b6d90e-kube-api-access-f45sf\") pod \"payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj\" (UID: \"1d67a8d4-e44a-4604-b653-fa7cd1b6d90e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj" Sep 16 13:47:10.129796 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.129775 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj" Sep 16 13:47:10.160818 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.160780 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt" Sep 16 13:47:10.409003 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.408930 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj"] Sep 16 13:47:10.411200 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:47:10.411164 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d67a8d4_e44a_4604_b653_fa7cd1b6d90e.slice/crio-b9039b384d95089ab3dc110e36a2415caf3e7fe2a0728a4e6845d222a0914134 WatchSource:0}: Error finding container b9039b384d95089ab3dc110e36a2415caf3e7fe2a0728a4e6845d222a0914134: Status 404 returned error can't find the container with id b9039b384d95089ab3dc110e36a2415caf3e7fe2a0728a4e6845d222a0914134 Sep 16 13:47:10.447078 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.447011 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt"] Sep 16 13:47:10.449229 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:47:10.449207 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb402416_c8de_4346_83fb_5997122a7272.slice/crio-89cc0d9a2f50f7bd5de2117da872c10d402025aae0f9acc756ccf4e0f43eba62 WatchSource:0}: Error finding container 89cc0d9a2f50f7bd5de2117da872c10d402025aae0f9acc756ccf4e0f43eba62: Status 404 returned error can't find the container with id 89cc0d9a2f50f7bd5de2117da872c10d402025aae0f9acc756ccf4e0f43eba62 Sep 16 13:47:10.785208 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.785165 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt" event={"ID":"db402416-c8de-4346-83fb-5997122a7272","Type":"ContainerStarted","Data":"14234db348fffbdca4da5b3340b8ec7e87b79647d3ae87bb6df88ad368b2e08c"} Sep 16 13:47:10.785422 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.785230 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt" event={"ID":"db402416-c8de-4346-83fb-5997122a7272","Type":"ContainerStarted","Data":"89cc0d9a2f50f7bd5de2117da872c10d402025aae0f9acc756ccf4e0f43eba62"} Sep 16 13:47:10.785422 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.785260 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt" Sep 16 13:47:10.786840 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.786818 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj" event={"ID":"1d67a8d4-e44a-4604-b653-fa7cd1b6d90e","Type":"ContainerStarted","Data":"507d990355d71e865e3ce4b0d458f139e947f15c350ef95bf59875285d71d266"} Sep 16 13:47:10.786975 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.786855 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj" event={"ID":"1d67a8d4-e44a-4604-b653-fa7cd1b6d90e","Type":"ContainerStarted","Data":"b9039b384d95089ab3dc110e36a2415caf3e7fe2a0728a4e6845d222a0914134"} Sep 16 13:47:10.786975 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.786943 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj" Sep 16 13:47:10.805350 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.805293 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt" podStartSLOduration=1.805278163 podStartE2EDuration="1.805278163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:47:10.803648967 +0000 UTC m=+978.904461672" watchObservedRunningTime="2026-09-16 13:47:10.805278163 +0000 UTC m=+978.906090877" Sep 16 13:47:10.823073 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:10.823022 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj" podStartSLOduration=1.823004895 podStartE2EDuration="1.823004895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:47:10.821710746 +0000 UTC m=+978.922523461" watchObservedRunningTime="2026-09-16 13:47:10.823004895 +0000 UTC m=+978.923817647" Sep 16 13:47:15.458196 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:15.458122 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:15.565033 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:15.565004 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:17.287245 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.287206 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25"] Sep 16 13:47:17.290770 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.290750 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:17.294376 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.294340 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4bb29c\"/\"premium-w4-4bb29c-kserve-self-signed-certs\"" Sep 16 13:47:17.324928 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.324894 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25"] Sep 16 13:47:17.368972 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.368938 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-model-cache\") pod \"premium-w4-4bb29c-kserve-7994c56ff4-74f25\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:17.369179 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.369007 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-home\") pod \"premium-w4-4bb29c-kserve-7994c56ff4-74f25\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:17.369179 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.369035 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-tls-certs\") pod \"premium-w4-4bb29c-kserve-7994c56ff4-74f25\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:17.369179 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.369053 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-tmp-dir\") pod \"premium-w4-4bb29c-kserve-7994c56ff4-74f25\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:17.369179 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.369099 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgvth\" (UniqueName: \"kubernetes.io/projected/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-kube-api-access-tgvth\") pod \"premium-w4-4bb29c-kserve-7994c56ff4-74f25\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:17.369179 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.369135 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-dshm\") pod \"premium-w4-4bb29c-kserve-7994c56ff4-74f25\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:17.384184 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.384138 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:17.470335 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.470296 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-home\") pod \"premium-w4-4bb29c-kserve-7994c56ff4-74f25\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:17.470335 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.470339 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-tls-certs\") pod \"premium-w4-4bb29c-kserve-7994c56ff4-74f25\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:17.470576 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.470358 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-tmp-dir\") pod \"premium-w4-4bb29c-kserve-7994c56ff4-74f25\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:17.470576 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.470401 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgvth\" (UniqueName: \"kubernetes.io/projected/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-kube-api-access-tgvth\") pod \"premium-w4-4bb29c-kserve-7994c56ff4-74f25\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:17.470576 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.470431 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-dshm\") pod \"premium-w4-4bb29c-kserve-7994c56ff4-74f25\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:17.470576 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.470448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-model-cache\") pod \"premium-w4-4bb29c-kserve-7994c56ff4-74f25\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:17.470839 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.470780 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-home\") pod \"premium-w4-4bb29c-kserve-7994c56ff4-74f25\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:17.470839 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.470801 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-model-cache\") pod \"premium-w4-4bb29c-kserve-7994c56ff4-74f25\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:17.470950 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.470836 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-tmp-dir\") pod \"premium-w4-4bb29c-kserve-7994c56ff4-74f25\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:17.472861 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.472836 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-dshm\") pod \"premium-w4-4bb29c-kserve-7994c56ff4-74f25\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:17.473188 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.473170 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-tls-certs\") pod \"premium-w4-4bb29c-kserve-7994c56ff4-74f25\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:17.483196 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.483167 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgvth\" (UniqueName: \"kubernetes.io/projected/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-kube-api-access-tgvth\") pod \"premium-w4-4bb29c-kserve-7994c56ff4-74f25\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:17.601635 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.601518 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:17.879543 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:17.879515 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25"] Sep 16 13:47:17.881664 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:47:17.881642 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fc23d93_10d4_4a71_ac2d_f9d881f008e3.slice/crio-1b43e365fbeca3f8a0fb33d2bb3ca10d38123d51c58223c5ab94ed4f5c7d83cb WatchSource:0}: Error finding container 1b43e365fbeca3f8a0fb33d2bb3ca10d38123d51c58223c5ab94ed4f5c7d83cb: Status 404 returned error can't find the container with id 1b43e365fbeca3f8a0fb33d2bb3ca10d38123d51c58223c5ab94ed4f5c7d83cb Sep 16 13:47:18.828752 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:18.828709 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" event={"ID":"9fc23d93-10d4-4a71-ac2d-f9d881f008e3","Type":"ContainerStarted","Data":"3d4a5bd04bb33281e5dd8ad6b7119461cfeea33c667337c67b2419c1039ad104"} Sep 16 13:47:18.829151 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:18.828763 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" event={"ID":"9fc23d93-10d4-4a71-ac2d-f9d881f008e3","Type":"ContainerStarted","Data":"1b43e365fbeca3f8a0fb33d2bb3ca10d38123d51c58223c5ab94ed4f5c7d83cb"} Sep 16 13:47:18.846254 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:18.846205 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" podStartSLOduration=1.846186455 podStartE2EDuration="1.846186455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:47:18.845714595 +0000 UTC m=+986.946527310" watchObservedRunningTime="2026-09-16 13:47:18.846186455 +0000 UTC m=+986.946999183" Sep 16 13:47:21.795208 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:21.795176 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt" Sep 16 13:47:21.795587 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:21.795244 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj" Sep 16 13:47:26.911650 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:26.911591 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:27.602158 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:27.602123 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:27.602349 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:27.602177 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:27.614581 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:27.614551 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:27.887541 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:27.887450 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:47:30.110669 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:30.110631 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:47:31.395138 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.395107 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:31.591918 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.591882 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn"] Sep 16 13:47:31.595543 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.595525 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:31.598251 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.598228 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1e5db0\"/\"default-dockercfg-jb6z7\"" Sep 16 13:47:31.598361 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.598276 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1e5db0\"/\"openshift-service-ca.crt\"" Sep 16 13:47:31.598361 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.598321 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1e5db0\"/\"kube-root-ca.crt\"" Sep 16 13:47:31.598361 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.598351 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1e5db0\"/\"test-model-1e5db0-kserve-self-signed-certs\"" Sep 16 13:47:31.608272 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.608248 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn"] Sep 16 13:47:31.675097 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.675005 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:31.708410 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.708388 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b09a78-25ab-4656-9261-49076035b91c-tls-certs\") pod \"test-model-1e5db0-kserve-647b747-wzvxn\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:31.708533 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.708427 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-dshm\") pod \"test-model-1e5db0-kserve-647b747-wzvxn\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:31.708533 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.708463 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-model-cache\") pod \"test-model-1e5db0-kserve-647b747-wzvxn\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:31.708533 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.708525 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-home\") pod \"test-model-1e5db0-kserve-647b747-wzvxn\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:31.708685 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.708576 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-tmp-dir\") pod \"test-model-1e5db0-kserve-647b747-wzvxn\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:31.708685 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.708638 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnzqd\" (UniqueName: \"kubernetes.io/projected/82b09a78-25ab-4656-9261-49076035b91c-kube-api-access-fnzqd\") pod \"test-model-1e5db0-kserve-647b747-wzvxn\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:31.810034 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.809999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-home\") pod \"test-model-1e5db0-kserve-647b747-wzvxn\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:31.810209 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.810040 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-tmp-dir\") pod \"test-model-1e5db0-kserve-647b747-wzvxn\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:31.810209 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.810174 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnzqd\" (UniqueName: \"kubernetes.io/projected/82b09a78-25ab-4656-9261-49076035b91c-kube-api-access-fnzqd\") pod \"test-model-1e5db0-kserve-647b747-wzvxn\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:31.810336 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.810285 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b09a78-25ab-4656-9261-49076035b91c-tls-certs\") pod \"test-model-1e5db0-kserve-647b747-wzvxn\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:31.810396 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.810337 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-dshm\") pod \"test-model-1e5db0-kserve-647b747-wzvxn\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:31.810396 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.810376 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-tmp-dir\") pod \"test-model-1e5db0-kserve-647b747-wzvxn\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:31.810396 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.810385 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-home\") pod \"test-model-1e5db0-kserve-647b747-wzvxn\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:31.810543 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.810399 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-model-cache\") pod \"test-model-1e5db0-kserve-647b747-wzvxn\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:31.810754 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.810729 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-model-cache\") pod \"test-model-1e5db0-kserve-647b747-wzvxn\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:31.812812 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.812780 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-dshm\") pod \"test-model-1e5db0-kserve-647b747-wzvxn\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:31.813100 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.813084 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b09a78-25ab-4656-9261-49076035b91c-tls-certs\") pod \"test-model-1e5db0-kserve-647b747-wzvxn\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:31.819682 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.819570 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnzqd\" (UniqueName: \"kubernetes.io/projected/82b09a78-25ab-4656-9261-49076035b91c-kube-api-access-fnzqd\") pod \"test-model-1e5db0-kserve-647b747-wzvxn\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:31.906898 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:31.906852 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:32.179330 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:32.179305 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn"] Sep 16 13:47:32.181347 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:47:32.181317 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82b09a78_25ab_4656_9261_49076035b91c.slice/crio-b3772a3ce8a73864e83e378c7fd34cf809295f05dd852508db2b3f5f028d1a24 WatchSource:0}: Error finding container b3772a3ce8a73864e83e378c7fd34cf809295f05dd852508db2b3f5f028d1a24: Status 404 returned error can't find the container with id b3772a3ce8a73864e83e378c7fd34cf809295f05dd852508db2b3f5f028d1a24 Sep 16 13:47:32.237203 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:32.237179 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:32.903785 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:32.903751 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" event={"ID":"82b09a78-25ab-4656-9261-49076035b91c","Type":"ContainerStarted","Data":"9cdb7df397b882a1b7df82e8df04cfe003deaeb36d9911b658f92bc42fae4115"} Sep 16 13:47:32.904163 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:32.903792 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" event={"ID":"82b09a78-25ab-4656-9261-49076035b91c","Type":"ContainerStarted","Data":"b3772a3ce8a73864e83e378c7fd34cf809295f05dd852508db2b3f5f028d1a24"} Sep 16 13:47:32.918453 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:32.918428 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:32.922584 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:32.922545 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" podStartSLOduration=1.9225314500000001 podStartE2EDuration="1.92253145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:47:32.922121135 +0000 UTC m=+1001.022933862" watchObservedRunningTime="2026-09-16 13:47:32.92253145 +0000 UTC m=+1001.023344164" Sep 16 13:47:33.019003 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:33.018972 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:33.120744 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:33.120702 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:34.026358 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:34.026323 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:34.454003 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:34.453925 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:35.076534 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:35.076502 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:35.741994 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:35.741960 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:35.891373 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:35.891335 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:41.907072 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:41.907033 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:41.907529 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:41.907118 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:41.920018 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:41.919996 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:41.965003 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:41.964973 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:47:49.027766 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:49.027726 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:51.759045 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:51.759013 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:54.871836 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:54.871796 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:54.927518 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:54.927487 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:55.015862 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:55.015833 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:55.114898 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:55.114863 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:55.613842 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:47:55.613807 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:48:05.590229 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:05.590189 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:06.428955 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:06.428918 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:12.653828 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:12.653787 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:12.767555 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:12.767464 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:12.855109 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:12.855076 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:14.019439 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:14.019406 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:14.120092 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:14.120056 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:15.346924 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:15.346889 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:16.872703 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:16.872671 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:16.973167 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:16.973135 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:21.335928 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:21.335897 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:21.431445 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:21.431413 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:22.025060 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:22.025024 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:22.902320 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:22.902283 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:48:27.968458 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:27.968423 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:28.065428 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:28.065395 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:28.205700 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:28.205669 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:28.594904 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:28.594868 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:28.737716 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:28.737680 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:31.416764 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:31.416726 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:32.111313 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:32.111216 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:32.206772 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:32.206742 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:32.324534 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:32.324501 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:33.252574 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:33.252541 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:33.930118 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:33.930086 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:34.834250 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:34.834219 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:35.509008 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:35.508962 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:36.352415 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:36.352378 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:36.421626 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:36.421580 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:36.681281 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:36.681197 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:36.883869 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:36.883836 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:36.984588 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:36.984548 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:37.063577 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:37.063547 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp"] Sep 16 13:48:37.067133 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:37.067115 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp" Sep 16 13:48:37.070641 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:37.070581 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ab3aa7c0\"" Sep 16 13:48:37.070748 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:37.070700 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ab3aa7c0-dockercfg-s847v\"" Sep 16 13:48:37.083842 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:37.083807 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp"] Sep 16 13:48:37.136168 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:37.136130 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ef29df-6c3a-4c73-bfcf-d8b857dcb027-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp\" (UID: \"13ef29df-6c3a-4c73-bfcf-d8b857dcb027\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp" Sep 16 13:48:37.136347 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:37.136192 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwskp\" (UniqueName: \"kubernetes.io/projected/13ef29df-6c3a-4c73-bfcf-d8b857dcb027-kube-api-access-vwskp\") pod \"payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp\" (UID: \"13ef29df-6c3a-4c73-bfcf-d8b857dcb027\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp" Sep 16 13:48:37.237507 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:37.237419 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ef29df-6c3a-4c73-bfcf-d8b857dcb027-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp\" (UID: \"13ef29df-6c3a-4c73-bfcf-d8b857dcb027\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp" Sep 16 13:48:37.237507 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:37.237474 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwskp\" (UniqueName: \"kubernetes.io/projected/13ef29df-6c3a-4c73-bfcf-d8b857dcb027-kube-api-access-vwskp\") pod \"payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp\" (UID: \"13ef29df-6c3a-4c73-bfcf-d8b857dcb027\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp" Sep 16 13:48:37.238312 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:37.238289 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ef29df-6c3a-4c73-bfcf-d8b857dcb027-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp\" (UID: \"13ef29df-6c3a-4c73-bfcf-d8b857dcb027\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp" Sep 16 13:48:37.247717 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:37.247686 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwskp\" (UniqueName: \"kubernetes.io/projected/13ef29df-6c3a-4c73-bfcf-d8b857dcb027-kube-api-access-vwskp\") pod \"payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp\" (UID: \"13ef29df-6c3a-4c73-bfcf-d8b857dcb027\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp" Sep 16 13:48:37.378739 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:37.378697 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp" Sep 16 13:48:37.668032 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:37.668006 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp"] Sep 16 13:48:37.669613 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:48:37.669576 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ef29df_6c3a_4c73_bfcf_d8b857dcb027.slice/crio-4cbcf226260111b6562c303f3b1869aede15b0b9264c7c85c4bb76b0f5a31db0 WatchSource:0}: Error finding container 4cbcf226260111b6562c303f3b1869aede15b0b9264c7c85c4bb76b0f5a31db0: Status 404 returned error can't find the container with id 4cbcf226260111b6562c303f3b1869aede15b0b9264c7c85c4bb76b0f5a31db0 Sep 16 13:48:38.246207 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:38.246176 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp" event={"ID":"13ef29df-6c3a-4c73-bfcf-d8b857dcb027","Type":"ContainerStarted","Data":"6dc71f1927f80f29e985b36bb564af19cff17d965853aa4689777d878802a96c"} Sep 16 13:48:38.246385 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:38.246228 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp" event={"ID":"13ef29df-6c3a-4c73-bfcf-d8b857dcb027","Type":"ContainerStarted","Data":"4cbcf226260111b6562c303f3b1869aede15b0b9264c7c85c4bb76b0f5a31db0"} Sep 16 13:48:38.246385 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:38.246340 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp" Sep 16 13:48:38.270181 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:38.270123 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp" podStartSLOduration=1.270107029 podStartE2EDuration="1.270107029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:48:38.268085691 +0000 UTC m=+1066.368898418" watchObservedRunningTime="2026-09-16 13:48:38.270107029 +0000 UTC m=+1066.370919743" Sep 16 13:48:44.569585 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.569505 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4"] Sep 16 13:48:44.573081 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.573065 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:44.576744 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.576720 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-4678b2\"/\"explicit-model-4678b2-kserve-self-signed-certs\"" Sep 16 13:48:44.576879 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.576726 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-4678b2\"/\"openshift-service-ca.crt\"" Sep 16 13:48:44.576879 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.576764 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-4678b2\"/\"kube-root-ca.crt\"" Sep 16 13:48:44.576879 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.576859 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-4678b2\"/\"default-dockercfg-tdn9s\"" Sep 16 13:48:44.582415 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.582388 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4"] Sep 16 13:48:44.603157 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.603124 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-dshm\") pod \"explicit-model-4678b2-kserve-64b4dd76cf-ddvf4\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:44.603316 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.603225 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fmgt\" (UniqueName: \"kubernetes.io/projected/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-kube-api-access-8fmgt\") pod \"explicit-model-4678b2-kserve-64b4dd76cf-ddvf4\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:44.603316 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.603305 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-tls-certs\") pod \"explicit-model-4678b2-kserve-64b4dd76cf-ddvf4\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:44.603408 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.603369 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-tmp-dir\") pod \"explicit-model-4678b2-kserve-64b4dd76cf-ddvf4\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:44.603408 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.603399 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-model-cache\") pod \"explicit-model-4678b2-kserve-64b4dd76cf-ddvf4\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:44.603488 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.603470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-home\") pod \"explicit-model-4678b2-kserve-64b4dd76cf-ddvf4\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:44.668407 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.668364 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:44.704446 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.704405 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-dshm\") pod \"explicit-model-4678b2-kserve-64b4dd76cf-ddvf4\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:44.704682 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.704476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fmgt\" (UniqueName: \"kubernetes.io/projected/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-kube-api-access-8fmgt\") pod \"explicit-model-4678b2-kserve-64b4dd76cf-ddvf4\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:44.704682 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.704536 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-tls-certs\") pod \"explicit-model-4678b2-kserve-64b4dd76cf-ddvf4\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:44.704682 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.704572 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-tmp-dir\") pod \"explicit-model-4678b2-kserve-64b4dd76cf-ddvf4\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:44.704682 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.704600 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-model-cache\") pod \"explicit-model-4678b2-kserve-64b4dd76cf-ddvf4\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:44.704682 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.704672 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-home\") pod \"explicit-model-4678b2-kserve-64b4dd76cf-ddvf4\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:44.705017 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.704998 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-tmp-dir\") pod \"explicit-model-4678b2-kserve-64b4dd76cf-ddvf4\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:44.705079 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.705028 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-model-cache\") pod \"explicit-model-4678b2-kserve-64b4dd76cf-ddvf4\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:44.705079 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.705064 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-home\") pod \"explicit-model-4678b2-kserve-64b4dd76cf-ddvf4\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:44.706920 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.706897 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-dshm\") pod \"explicit-model-4678b2-kserve-64b4dd76cf-ddvf4\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:44.707174 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.707149 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-tls-certs\") pod \"explicit-model-4678b2-kserve-64b4dd76cf-ddvf4\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:44.712727 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.712703 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fmgt\" (UniqueName: \"kubernetes.io/projected/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-kube-api-access-8fmgt\") pod \"explicit-model-4678b2-kserve-64b4dd76cf-ddvf4\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:44.883381 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:44.883289 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:45.160817 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:45.160791 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4"] Sep 16 13:48:45.163050 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:48:45.163018 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cb77e44_d96f_4b4b_97f5_6f71025e5a85.slice/crio-fa495d32b2e6d6896e26c686bc603589f41ceb61bfbdf09c7bb2e647bd7dd328 WatchSource:0}: Error finding container fa495d32b2e6d6896e26c686bc603589f41ceb61bfbdf09c7bb2e647bd7dd328: Status 404 returned error can't find the container with id fa495d32b2e6d6896e26c686bc603589f41ceb61bfbdf09c7bb2e647bd7dd328 Sep 16 13:48:45.285557 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:45.285519 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" event={"ID":"2cb77e44-d96f-4b4b-97f5-6f71025e5a85","Type":"ContainerStarted","Data":"863bc62fb770752aa84378b2ea639a0fa14af7be0aa10fc52dd67aa4c8cd9b3a"} Sep 16 13:48:45.285776 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:45.285568 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" event={"ID":"2cb77e44-d96f-4b4b-97f5-6f71025e5a85","Type":"ContainerStarted","Data":"fa495d32b2e6d6896e26c686bc603589f41ceb61bfbdf09c7bb2e647bd7dd328"} Sep 16 13:48:45.305124 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:45.305073 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" podStartSLOduration=1.305054929 podStartE2EDuration="1.305054929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:48:45.303854077 +0000 UTC m=+1073.404666791" watchObservedRunningTime="2026-09-16 13:48:45.305054929 +0000 UTC m=+1073.405867643" Sep 16 13:48:49.253696 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:49.253661 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp" Sep 16 13:48:54.206577 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:54.206538 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:48:54.884133 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:54.884098 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:54.884360 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:54.884197 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:54.896612 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:54.896585 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:55.350296 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:55.350266 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:48:57.469490 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.469449 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx"] Sep 16 13:48:57.473197 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.473175 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:48:57.475947 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.475929 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4bb29c\"/\"e2e-conflict-a-08a980e5-kserve-self-signed-certs\"" Sep 16 13:48:57.484219 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.484197 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx"] Sep 16 13:48:57.531202 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.531167 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdv7w\" (UniqueName: \"kubernetes.io/projected/a3bde040-bc64-44a8-98b7-9abe14f6215c-kube-api-access-jdv7w\") pod \"e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:48:57.531202 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.531205 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-tmp-dir\") pod \"e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:48:57.531410 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.531229 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-dshm\") pod \"e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:48:57.531410 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.531298 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3bde040-bc64-44a8-98b7-9abe14f6215c-tls-certs\") pod \"e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:48:57.531410 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.531367 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-home\") pod \"e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:48:57.531504 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.531412 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-model-cache\") pod \"e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:48:57.632444 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.632404 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3bde040-bc64-44a8-98b7-9abe14f6215c-tls-certs\") pod \"e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:48:57.632680 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.632485 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-home\") pod \"e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:48:57.632680 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.632522 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-model-cache\") pod \"e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:48:57.632680 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.632568 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdv7w\" (UniqueName: \"kubernetes.io/projected/a3bde040-bc64-44a8-98b7-9abe14f6215c-kube-api-access-jdv7w\") pod \"e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:48:57.632680 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.632590 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-tmp-dir\") pod \"e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:48:57.632680 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.632645 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-dshm\") pod \"e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:48:57.632981 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.632957 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-home\") pod \"e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:48:57.633038 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.633012 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-model-cache\") pod \"e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:48:57.633084 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.633054 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-tmp-dir\") pod \"e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:48:57.635085 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.635058 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-dshm\") pod \"e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:48:57.635352 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.635329 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3bde040-bc64-44a8-98b7-9abe14f6215c-tls-certs\") pod \"e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:48:57.640891 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.640870 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdv7w\" (UniqueName: \"kubernetes.io/projected/a3bde040-bc64-44a8-98b7-9abe14f6215c-kube-api-access-jdv7w\") pod \"e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:48:57.777490 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.777391 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:57.783358 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:57.783338 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:48:58.057186 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:58.057159 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx"] Sep 16 13:48:58.059135 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:48:58.059112 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3bde040_bc64_44a8_98b7_9abe14f6215c.slice/crio-90d2ae653cd4a2fb382d40728658d3a2c4096b1514277901fdf8b8e5739e9a39 WatchSource:0}: Error finding container 90d2ae653cd4a2fb382d40728658d3a2c4096b1514277901fdf8b8e5739e9a39: Status 404 returned error can't find the container with id 90d2ae653cd4a2fb382d40728658d3a2c4096b1514277901fdf8b8e5739e9a39 Sep 16 13:48:58.359564 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:58.359473 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" event={"ID":"a3bde040-bc64-44a8-98b7-9abe14f6215c","Type":"ContainerStarted","Data":"98687ff24ee533463b1f74e608c92849586653efb50ce052e3dfe5ecd730855a"} Sep 16 13:48:58.359564 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:58.359516 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" event={"ID":"a3bde040-bc64-44a8-98b7-9abe14f6215c","Type":"ContainerStarted","Data":"90d2ae653cd4a2fb382d40728658d3a2c4096b1514277901fdf8b8e5739e9a39"} Sep 16 13:48:58.376968 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:58.376921 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" podStartSLOduration=1.376905172 podStartE2EDuration="1.376905172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:48:58.375523166 +0000 UTC m=+1086.476335879" watchObservedRunningTime="2026-09-16 13:48:58.376905172 +0000 UTC m=+1086.477717925" Sep 16 13:48:58.720367 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:58.720310 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:59.673054 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:48:59.673019 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:00.445349 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:00.445319 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:01.166384 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:01.166351 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:02.048949 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:02.048914 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:02.201952 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:02.201920 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:02.740292 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:02.740254 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:02.860012 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:02.859974 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:02.951085 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:02.951055 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:03.276940 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:03.276905 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:03.701832 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:03.701742 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:07.783632 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:07.783568 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:49:07.784175 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:07.783645 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:49:07.796210 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:07.796188 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:49:08.428411 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:08.428382 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:49:09.208385 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:09.208353 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:09.772309 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:09.772277 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:09.827896 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:09.827859 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:10.843895 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:10.843854 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:11.677090 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:11.677056 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4"] Sep 16 13:49:11.707473 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:11.707429 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:11.801788 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:11.801757 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:11.807508 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:11.807477 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4678b2-2p8g9"] Sep 16 13:49:11.811288 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:11.811270 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4678b2-2p8g9" Sep 16 13:49:11.813848 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:11.813828 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:49:11.813962 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:11.813828 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-thk5m\"" Sep 16 13:49:11.819266 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:11.819229 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4678b2-2p8g9"] Sep 16 13:49:11.975376 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:11.975344 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/588ea072-823f-4eed-a7ef-626a3a130af5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4678b2-2p8g9\" (UID: \"588ea072-823f-4eed-a7ef-626a3a130af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4678b2-2p8g9" Sep 16 13:49:12.076754 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:12.076697 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/588ea072-823f-4eed-a7ef-626a3a130af5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4678b2-2p8g9\" (UID: \"588ea072-823f-4eed-a7ef-626a3a130af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4678b2-2p8g9" Sep 16 13:49:12.077327 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:12.077308 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/588ea072-823f-4eed-a7ef-626a3a130af5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4678b2-2p8g9\" (UID: \"588ea072-823f-4eed-a7ef-626a3a130af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4678b2-2p8g9" Sep 16 13:49:12.124120 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:12.124092 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4678b2-2p8g9" Sep 16 13:49:12.393563 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:12.393536 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4678b2-2p8g9"] Sep 16 13:49:12.395776 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:49:12.395743 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod588ea072_823f_4eed_a7ef_626a3a130af5.slice/crio-8c2a932b9656742a03a95f2bec21bf7394f5fe26d61f287d94a286ca7da8dbac WatchSource:0}: Error finding container 8c2a932b9656742a03a95f2bec21bf7394f5fe26d61f287d94a286ca7da8dbac: Status 404 returned error can't find the container with id 8c2a932b9656742a03a95f2bec21bf7394f5fe26d61f287d94a286ca7da8dbac Sep 16 13:49:12.438556 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:12.438526 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4678b2-2p8g9" event={"ID":"588ea072-823f-4eed-a7ef-626a3a130af5","Type":"ContainerStarted","Data":"8c2a932b9656742a03a95f2bec21bf7394f5fe26d61f287d94a286ca7da8dbac"} Sep 16 13:49:12.627229 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:12.627202 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:13.424270 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:13.424172 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:13.444880 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:13.444852 2580 generic.go:356] "Generic (PLEG): container finished" podID="588ea072-823f-4eed-a7ef-626a3a130af5" containerID="a7b7812857f37731a2408061abeb37f1518f9116aef1fb39ba13194e725dec63" exitCode=0 Sep 16 13:49:13.445015 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:13.444942 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4678b2-2p8g9" event={"ID":"588ea072-823f-4eed-a7ef-626a3a130af5","Type":"ContainerDied","Data":"a7b7812857f37731a2408061abeb37f1518f9116aef1fb39ba13194e725dec63"} Sep 16 13:49:13.535936 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:13.535893 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:14.307937 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:14.307899 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:14.588655 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:14.588626 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4678b2-2p8g9" Sep 16 13:49:14.703915 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:14.703879 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/588ea072-823f-4eed-a7ef-626a3a130af5-maas-api-service-ca\") pod \"588ea072-823f-4eed-a7ef-626a3a130af5\" (UID: \"588ea072-823f-4eed-a7ef-626a3a130af5\") " Sep 16 13:49:14.704221 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:14.704198 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/588ea072-823f-4eed-a7ef-626a3a130af5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "588ea072-823f-4eed-a7ef-626a3a130af5" (UID: "588ea072-823f-4eed-a7ef-626a3a130af5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:49:14.805210 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:14.805174 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/588ea072-823f-4eed-a7ef-626a3a130af5-maas-api-service-ca\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:49:15.139100 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:15.139063 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:15.457149 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:15.457055 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4678b2-2p8g9" event={"ID":"588ea072-823f-4eed-a7ef-626a3a130af5","Type":"ContainerDied","Data":"8c2a932b9656742a03a95f2bec21bf7394f5fe26d61f287d94a286ca7da8dbac"} Sep 16 13:49:15.457149 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:15.457112 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c2a932b9656742a03a95f2bec21bf7394f5fe26d61f287d94a286ca7da8dbac" Sep 16 13:49:15.457149 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:15.457065 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4678b2-2p8g9" Sep 16 13:49:15.925162 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:15.925127 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:16.702145 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:16.702106 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:17.626307 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:17.626274 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:18.430545 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:18.430513 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:20.115340 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:20.115295 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:20.701536 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:20.701505 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:20.730106 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:20.730078 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp"] Sep 16 13:49:20.730303 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:20.730277 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp" podUID="13ef29df-6c3a-4c73-bfcf-d8b857dcb027" containerName="payload-pre-processing" containerID="cri-o://6dc71f1927f80f29e985b36bb564af19cff17d965853aa4689777d878802a96c" gracePeriod=30 Sep 16 13:49:20.774534 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:20.774489 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:20.878685 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:20.878639 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:20.951917 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:20.951837 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:20.997092 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:20.997064 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp" Sep 16 13:49:21.060137 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:21.060100 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwskp\" (UniqueName: \"kubernetes.io/projected/13ef29df-6c3a-4c73-bfcf-d8b857dcb027-kube-api-access-vwskp\") pod \"13ef29df-6c3a-4c73-bfcf-d8b857dcb027\" (UID: \"13ef29df-6c3a-4c73-bfcf-d8b857dcb027\") " Sep 16 13:49:21.060331 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:21.060146 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ef29df-6c3a-4c73-bfcf-d8b857dcb027-plugins-config-volume\") pod \"13ef29df-6c3a-4c73-bfcf-d8b857dcb027\" (UID: \"13ef29df-6c3a-4c73-bfcf-d8b857dcb027\") " Sep 16 13:49:21.060530 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:21.060508 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ef29df-6c3a-4c73-bfcf-d8b857dcb027-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13ef29df-6c3a-4c73-bfcf-d8b857dcb027" (UID: "13ef29df-6c3a-4c73-bfcf-d8b857dcb027"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:49:21.062377 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:21.062351 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ef29df-6c3a-4c73-bfcf-d8b857dcb027-kube-api-access-vwskp" (OuterVolumeSpecName: "kube-api-access-vwskp") pod "13ef29df-6c3a-4c73-bfcf-d8b857dcb027" (UID: "13ef29df-6c3a-4c73-bfcf-d8b857dcb027"). InnerVolumeSpecName "kube-api-access-vwskp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:49:21.161659 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:21.161616 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwskp\" (UniqueName: \"kubernetes.io/projected/13ef29df-6c3a-4c73-bfcf-d8b857dcb027-kube-api-access-vwskp\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:49:21.161659 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:21.161643 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ef29df-6c3a-4c73-bfcf-d8b857dcb027-plugins-config-volume\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:49:21.171677 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:21.171648 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:21.279348 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:21.279313 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:21.414204 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:21.414171 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:49:21.488515 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:21.488479 2580 generic.go:356] "Generic (PLEG): container finished" podID="13ef29df-6c3a-4c73-bfcf-d8b857dcb027" containerID="6dc71f1927f80f29e985b36bb564af19cff17d965853aa4689777d878802a96c" exitCode=0 Sep 16 13:49:21.488707 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:21.488539 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp" Sep 16 13:49:21.488707 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:21.488560 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp" event={"ID":"13ef29df-6c3a-4c73-bfcf-d8b857dcb027","Type":"ContainerDied","Data":"6dc71f1927f80f29e985b36bb564af19cff17d965853aa4689777d878802a96c"} Sep 16 13:49:21.488707 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:21.488630 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp" event={"ID":"13ef29df-6c3a-4c73-bfcf-d8b857dcb027","Type":"ContainerDied","Data":"4cbcf226260111b6562c303f3b1869aede15b0b9264c7c85c4bb76b0f5a31db0"} Sep 16 13:49:21.488707 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:21.488660 2580 scope.go:117] "RemoveContainer" containerID="6dc71f1927f80f29e985b36bb564af19cff17d965853aa4689777d878802a96c" Sep 16 13:49:21.497995 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:21.497972 2580 scope.go:117] "RemoveContainer" containerID="6dc71f1927f80f29e985b36bb564af19cff17d965853aa4689777d878802a96c" Sep 16 13:49:21.498267 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:49:21.498248 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc71f1927f80f29e985b36bb564af19cff17d965853aa4689777d878802a96c\": container with ID starting with 6dc71f1927f80f29e985b36bb564af19cff17d965853aa4689777d878802a96c not found: ID does not exist" containerID="6dc71f1927f80f29e985b36bb564af19cff17d965853aa4689777d878802a96c" Sep 16 13:49:21.498340 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:21.498273 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc71f1927f80f29e985b36bb564af19cff17d965853aa4689777d878802a96c"} err="failed to get container status \"6dc71f1927f80f29e985b36bb564af19cff17d965853aa4689777d878802a96c\": rpc error: code = NotFound desc = could not find container \"6dc71f1927f80f29e985b36bb564af19cff17d965853aa4689777d878802a96c\": container with ID starting with 6dc71f1927f80f29e985b36bb564af19cff17d965853aa4689777d878802a96c not found: ID does not exist" Sep 16 13:49:21.512536 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:21.512508 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp"] Sep 16 13:49:21.519184 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:21.519147 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ab3aa7c0-85858c885d-xxcpp"] Sep 16 13:49:22.034800 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:22.034763 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:22.355552 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:22.355475 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13ef29df-6c3a-4c73-bfcf-d8b857dcb027" path="/var/lib/kubelet/pods/13ef29df-6c3a-4c73-bfcf-d8b857dcb027/volumes" Sep 16 13:49:22.756747 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:22.756715 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:22.786241 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:22.786202 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt"] Sep 16 13:49:22.786488 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:22.786462 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt" podUID="db402416-c8de-4346-83fb-5997122a7272" containerName="payload-processing" containerID="cri-o://14234db348fffbdca4da5b3340b8ec7e87b79647d3ae87bb6df88ad368b2e08c" gracePeriod=30 Sep 16 13:49:22.794627 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:22.794586 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj"] Sep 16 13:49:22.794840 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:22.794816 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj" podUID="1d67a8d4-e44a-4604-b653-fa7cd1b6d90e" containerName="payload-pre-processing" containerID="cri-o://507d990355d71e865e3ce4b0d458f139e947f15c350ef95bf59875285d71d266" gracePeriod=30 Sep 16 13:49:22.832013 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:22.831978 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:22.894516 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:22.894459 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:22.975916 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:22.975885 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4678b2-2p8g9"] Sep 16 13:49:22.977320 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:22.977295 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:22.978507 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:22.978428 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4678b2-2p8g9"] Sep 16 13:49:23.044045 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.044017 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:23.045467 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.045444 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt" Sep 16 13:49:23.076667 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.076642 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj" Sep 16 13:49:23.081930 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.081909 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db402416-c8de-4346-83fb-5997122a7272-plugins-config-volume\") pod \"db402416-c8de-4346-83fb-5997122a7272\" (UID: \"db402416-c8de-4346-83fb-5997122a7272\") " Sep 16 13:49:23.082044 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.082026 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79rv5\" (UniqueName: \"kubernetes.io/projected/db402416-c8de-4346-83fb-5997122a7272-kube-api-access-79rv5\") pod \"db402416-c8de-4346-83fb-5997122a7272\" (UID: \"db402416-c8de-4346-83fb-5997122a7272\") " Sep 16 13:49:23.082275 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.082252 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db402416-c8de-4346-83fb-5997122a7272-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "db402416-c8de-4346-83fb-5997122a7272" (UID: "db402416-c8de-4346-83fb-5997122a7272"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:49:23.082359 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.082347 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db402416-c8de-4346-83fb-5997122a7272-plugins-config-volume\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:49:23.084338 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.084315 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db402416-c8de-4346-83fb-5997122a7272-kube-api-access-79rv5" (OuterVolumeSpecName: "kube-api-access-79rv5") pod "db402416-c8de-4346-83fb-5997122a7272" (UID: "db402416-c8de-4346-83fb-5997122a7272"). InnerVolumeSpecName "kube-api-access-79rv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:49:23.103812 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.103784 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:23.182826 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.182790 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d67a8d4-e44a-4604-b653-fa7cd1b6d90e-plugins-config-volume\") pod \"1d67a8d4-e44a-4604-b653-fa7cd1b6d90e\" (UID: \"1d67a8d4-e44a-4604-b653-fa7cd1b6d90e\") " Sep 16 13:49:23.182996 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.182874 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f45sf\" (UniqueName: \"kubernetes.io/projected/1d67a8d4-e44a-4604-b653-fa7cd1b6d90e-kube-api-access-f45sf\") pod \"1d67a8d4-e44a-4604-b653-fa7cd1b6d90e\" (UID: \"1d67a8d4-e44a-4604-b653-fa7cd1b6d90e\") " Sep 16 13:49:23.183115 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.183103 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79rv5\" (UniqueName: \"kubernetes.io/projected/db402416-c8de-4346-83fb-5997122a7272-kube-api-access-79rv5\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:49:23.183199 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.183177 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d67a8d4-e44a-4604-b653-fa7cd1b6d90e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d67a8d4-e44a-4604-b653-fa7cd1b6d90e" (UID: "1d67a8d4-e44a-4604-b653-fa7cd1b6d90e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:49:23.185069 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.185047 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d67a8d4-e44a-4604-b653-fa7cd1b6d90e-kube-api-access-f45sf" (OuterVolumeSpecName: "kube-api-access-f45sf") pod "1d67a8d4-e44a-4604-b653-fa7cd1b6d90e" (UID: "1d67a8d4-e44a-4604-b653-fa7cd1b6d90e"). InnerVolumeSpecName "kube-api-access-f45sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:49:23.284010 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.283923 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d67a8d4-e44a-4604-b653-fa7cd1b6d90e-plugins-config-volume\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:49:23.284010 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.283953 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f45sf\" (UniqueName: \"kubernetes.io/projected/1d67a8d4-e44a-4604-b653-fa7cd1b6d90e-kube-api-access-f45sf\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:49:23.346668 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.346634 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:23.502003 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.501970 2580 generic.go:356] "Generic (PLEG): container finished" podID="db402416-c8de-4346-83fb-5997122a7272" containerID="14234db348fffbdca4da5b3340b8ec7e87b79647d3ae87bb6df88ad368b2e08c" exitCode=0 Sep 16 13:49:23.502499 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.502029 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt" Sep 16 13:49:23.502499 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.502056 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt" event={"ID":"db402416-c8de-4346-83fb-5997122a7272","Type":"ContainerDied","Data":"14234db348fffbdca4da5b3340b8ec7e87b79647d3ae87bb6df88ad368b2e08c"} Sep 16 13:49:23.502499 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.502112 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt" event={"ID":"db402416-c8de-4346-83fb-5997122a7272","Type":"ContainerDied","Data":"89cc0d9a2f50f7bd5de2117da872c10d402025aae0f9acc756ccf4e0f43eba62"} Sep 16 13:49:23.502499 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.502137 2580 scope.go:117] "RemoveContainer" containerID="14234db348fffbdca4da5b3340b8ec7e87b79647d3ae87bb6df88ad368b2e08c" Sep 16 13:49:23.503737 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.503711 2580 generic.go:356] "Generic (PLEG): container finished" podID="1d67a8d4-e44a-4604-b653-fa7cd1b6d90e" containerID="507d990355d71e865e3ce4b0d458f139e947f15c350ef95bf59875285d71d266" exitCode=0 Sep 16 13:49:23.503873 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.503748 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj" Sep 16 13:49:23.503873 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.503795 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj" event={"ID":"1d67a8d4-e44a-4604-b653-fa7cd1b6d90e","Type":"ContainerDied","Data":"507d990355d71e865e3ce4b0d458f139e947f15c350ef95bf59875285d71d266"} Sep 16 13:49:23.503873 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.503843 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj" event={"ID":"1d67a8d4-e44a-4604-b653-fa7cd1b6d90e","Type":"ContainerDied","Data":"b9039b384d95089ab3dc110e36a2415caf3e7fe2a0728a4e6845d222a0914134"} Sep 16 13:49:23.511907 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.511880 2580 scope.go:117] "RemoveContainer" containerID="14234db348fffbdca4da5b3340b8ec7e87b79647d3ae87bb6df88ad368b2e08c" Sep 16 13:49:23.512184 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:49:23.512160 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14234db348fffbdca4da5b3340b8ec7e87b79647d3ae87bb6df88ad368b2e08c\": container with ID starting with 14234db348fffbdca4da5b3340b8ec7e87b79647d3ae87bb6df88ad368b2e08c not found: ID does not exist" containerID="14234db348fffbdca4da5b3340b8ec7e87b79647d3ae87bb6df88ad368b2e08c" Sep 16 13:49:23.512272 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.512195 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14234db348fffbdca4da5b3340b8ec7e87b79647d3ae87bb6df88ad368b2e08c"} err="failed to get container status \"14234db348fffbdca4da5b3340b8ec7e87b79647d3ae87bb6df88ad368b2e08c\": rpc error: code = NotFound desc = could not find container \"14234db348fffbdca4da5b3340b8ec7e87b79647d3ae87bb6df88ad368b2e08c\": container with ID starting with 14234db348fffbdca4da5b3340b8ec7e87b79647d3ae87bb6df88ad368b2e08c not found: ID does not exist" Sep 16 13:49:23.512272 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.512225 2580 scope.go:117] "RemoveContainer" containerID="507d990355d71e865e3ce4b0d458f139e947f15c350ef95bf59875285d71d266" Sep 16 13:49:23.520854 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.520840 2580 scope.go:117] "RemoveContainer" containerID="507d990355d71e865e3ce4b0d458f139e947f15c350ef95bf59875285d71d266" Sep 16 13:49:23.521132 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:49:23.521115 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"507d990355d71e865e3ce4b0d458f139e947f15c350ef95bf59875285d71d266\": container with ID starting with 507d990355d71e865e3ce4b0d458f139e947f15c350ef95bf59875285d71d266 not found: ID does not exist" containerID="507d990355d71e865e3ce4b0d458f139e947f15c350ef95bf59875285d71d266" Sep 16 13:49:23.521173 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.521140 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507d990355d71e865e3ce4b0d458f139e947f15c350ef95bf59875285d71d266"} err="failed to get container status \"507d990355d71e865e3ce4b0d458f139e947f15c350ef95bf59875285d71d266\": rpc error: code = NotFound desc = could not find container \"507d990355d71e865e3ce4b0d458f139e947f15c350ef95bf59875285d71d266\": container with ID starting with 507d990355d71e865e3ce4b0d458f139e947f15c350ef95bf59875285d71d266 not found: ID does not exist" Sep 16 13:49:23.529470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.529450 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj"] Sep 16 13:49:23.534993 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.534945 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4678b2-58686f4f8-l2xtj"] Sep 16 13:49:23.545707 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.545682 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt"] Sep 16 13:49:23.547612 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.547585 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4678b2-6d48479769-6k6wt"] Sep 16 13:49:23.826391 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:23.826360 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:24.362316 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:24.357558 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d67a8d4-e44a-4604-b653-fa7cd1b6d90e" path="/var/lib/kubelet/pods/1d67a8d4-e44a-4604-b653-fa7cd1b6d90e/volumes" Sep 16 13:49:24.362316 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:24.358130 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="588ea072-823f-4eed-a7ef-626a3a130af5" path="/var/lib/kubelet/pods/588ea072-823f-4eed-a7ef-626a3a130af5/volumes" Sep 16 13:49:24.362316 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:24.358711 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db402416-c8de-4346-83fb-5997122a7272" path="/var/lib/kubelet/pods/db402416-c8de-4346-83fb-5997122a7272/volumes" Sep 16 13:49:24.526123 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:24.526091 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:25.332714 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:25.332679 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:26.009812 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:26.009783 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:26.696708 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:26.696664 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" podUID="2cb77e44-d96f-4b4b-97f5-6f71025e5a85" containerName="main" containerID="cri-o://863bc62fb770752aa84378b2ea639a0fa14af7be0aa10fc52dd67aa4c8cd9b3a" gracePeriod=45 Sep 16 13:49:26.822380 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:26.822351 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:26.957907 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:26.957844 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:49:27.021231 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.021201 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-dshm\") pod \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " Sep 16 13:49:27.021709 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.021239 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fmgt\" (UniqueName: \"kubernetes.io/projected/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-kube-api-access-8fmgt\") pod \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " Sep 16 13:49:27.021709 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.021279 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-tls-certs\") pod \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " Sep 16 13:49:27.021709 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.021304 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-tmp-dir\") pod \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " Sep 16 13:49:27.021709 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.021400 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-home\") pod \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " Sep 16 13:49:27.021709 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.021516 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-model-cache\") pod \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\" (UID: \"2cb77e44-d96f-4b4b-97f5-6f71025e5a85\") " Sep 16 13:49:27.021709 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.021557 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2cb77e44-d96f-4b4b-97f5-6f71025e5a85" (UID: "2cb77e44-d96f-4b4b-97f5-6f71025e5a85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:49:27.021709 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.021676 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-home" (OuterVolumeSpecName: "home") pod "2cb77e44-d96f-4b4b-97f5-6f71025e5a85" (UID: "2cb77e44-d96f-4b4b-97f5-6f71025e5a85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:49:27.022254 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.021803 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-model-cache" (OuterVolumeSpecName: "model-cache") pod "2cb77e44-d96f-4b4b-97f5-6f71025e5a85" (UID: "2cb77e44-d96f-4b4b-97f5-6f71025e5a85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:49:27.022254 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.021929 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:49:27.022254 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.021948 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:49:27.022254 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.021957 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:49:27.023755 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.023732 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-dshm" (OuterVolumeSpecName: "dshm") pod "2cb77e44-d96f-4b4b-97f5-6f71025e5a85" (UID: "2cb77e44-d96f-4b4b-97f5-6f71025e5a85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:49:27.023850 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.023753 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-kube-api-access-8fmgt" (OuterVolumeSpecName: "kube-api-access-8fmgt") pod "2cb77e44-d96f-4b4b-97f5-6f71025e5a85" (UID: "2cb77e44-d96f-4b4b-97f5-6f71025e5a85"). InnerVolumeSpecName "kube-api-access-8fmgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:49:27.023850 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.023796 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2cb77e44-d96f-4b4b-97f5-6f71025e5a85" (UID: "2cb77e44-d96f-4b4b-97f5-6f71025e5a85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:49:27.123280 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.123244 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:49:27.123280 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.123273 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fmgt\" (UniqueName: \"kubernetes.io/projected/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-kube-api-access-8fmgt\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:49:27.123280 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.123283 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb77e44-d96f-4b4b-97f5-6f71025e5a85-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:49:27.527427 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.527390 2580 generic.go:356] "Generic (PLEG): container finished" podID="2cb77e44-d96f-4b4b-97f5-6f71025e5a85" containerID="863bc62fb770752aa84378b2ea639a0fa14af7be0aa10fc52dd67aa4c8cd9b3a" exitCode=0 Sep 16 13:49:27.527583 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.527434 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" Sep 16 13:49:27.527583 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.527477 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" event={"ID":"2cb77e44-d96f-4b4b-97f5-6f71025e5a85","Type":"ContainerDied","Data":"863bc62fb770752aa84378b2ea639a0fa14af7be0aa10fc52dd67aa4c8cd9b3a"} Sep 16 13:49:27.527583 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.527523 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4" event={"ID":"2cb77e44-d96f-4b4b-97f5-6f71025e5a85","Type":"ContainerDied","Data":"fa495d32b2e6d6896e26c686bc603589f41ceb61bfbdf09c7bb2e647bd7dd328"} Sep 16 13:49:27.527583 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.527542 2580 scope.go:117] "RemoveContainer" containerID="863bc62fb770752aa84378b2ea639a0fa14af7be0aa10fc52dd67aa4c8cd9b3a" Sep 16 13:49:27.537000 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.536981 2580 scope.go:117] "RemoveContainer" containerID="863bc62fb770752aa84378b2ea639a0fa14af7be0aa10fc52dd67aa4c8cd9b3a" Sep 16 13:49:27.537266 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:49:27.537247 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"863bc62fb770752aa84378b2ea639a0fa14af7be0aa10fc52dd67aa4c8cd9b3a\": container with ID starting with 863bc62fb770752aa84378b2ea639a0fa14af7be0aa10fc52dd67aa4c8cd9b3a not found: ID does not exist" containerID="863bc62fb770752aa84378b2ea639a0fa14af7be0aa10fc52dd67aa4c8cd9b3a" Sep 16 13:49:27.537321 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.537274 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"863bc62fb770752aa84378b2ea639a0fa14af7be0aa10fc52dd67aa4c8cd9b3a"} err="failed to get container status \"863bc62fb770752aa84378b2ea639a0fa14af7be0aa10fc52dd67aa4c8cd9b3a\": rpc error: code = NotFound desc = could not find container \"863bc62fb770752aa84378b2ea639a0fa14af7be0aa10fc52dd67aa4c8cd9b3a\": container with ID starting with 863bc62fb770752aa84378b2ea639a0fa14af7be0aa10fc52dd67aa4c8cd9b3a not found: ID does not exist" Sep 16 13:49:27.548729 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.548704 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4"] Sep 16 13:49:27.552470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.552438 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-4678b2/explicit-model-4678b2-kserve-64b4dd76cf-ddvf4"] Sep 16 13:49:27.704770 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.704711 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:27.804350 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.804242 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:27.976626 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:27.976572 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:28.317514 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:28.317473 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:28.355440 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:28.355412 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cb77e44-d96f-4b4b-97f5-6f71025e5a85" path="/var/lib/kubelet/pods/2cb77e44-d96f-4b4b-97f5-6f71025e5a85/volumes" Sep 16 13:49:28.409989 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:28.409960 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:28.537499 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:28.537465 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:29.317425 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:29.317392 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:29.419025 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:29.418990 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:29.602438 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:29.602350 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:30.052913 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:30.052884 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:30.355653 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:30.355550 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:30.908193 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:30.908161 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:31.904886 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:31.904819 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:32.003480 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:32.003440 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:32.103448 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:32.103349 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:34.435359 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:34.435316 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:34.538551 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:34.538501 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:34.635951 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:34.635923 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:34.799440 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:34.799409 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:40.471121 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:40.471088 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr"] Sep 16 13:49:40.492072 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:40.492041 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:41.011224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:41.011191 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:41.111091 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:41.111061 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:41.985848 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:41.985809 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:42.350977 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:42.350865 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:43.049726 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:43.049686 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:44.925031 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:44.924997 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:45.639816 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:45.639764 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:45.738183 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:45.738153 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:45.838807 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:45.838770 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:46.595242 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:46.595203 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:47.526134 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:47.526089 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:48.373188 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.373152 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp"] Sep 16 13:49:48.373566 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.373501 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d67a8d4-e44a-4604-b653-fa7cd1b6d90e" containerName="payload-pre-processing" Sep 16 13:49:48.373566 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.373514 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d67a8d4-e44a-4604-b653-fa7cd1b6d90e" containerName="payload-pre-processing" Sep 16 13:49:48.373566 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.373527 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db402416-c8de-4346-83fb-5997122a7272" containerName="payload-processing" Sep 16 13:49:48.373566 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.373532 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="db402416-c8de-4346-83fb-5997122a7272" containerName="payload-processing" Sep 16 13:49:48.373566 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.373547 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="588ea072-823f-4eed-a7ef-626a3a130af5" containerName="revoke-keys" Sep 16 13:49:48.373566 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.373552 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="588ea072-823f-4eed-a7ef-626a3a130af5" containerName="revoke-keys" Sep 16 13:49:48.373566 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.373564 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13ef29df-6c3a-4c73-bfcf-d8b857dcb027" containerName="payload-pre-processing" Sep 16 13:49:48.373566 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.373570 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="13ef29df-6c3a-4c73-bfcf-d8b857dcb027" containerName="payload-pre-processing" Sep 16 13:49:48.373846 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.373578 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cb77e44-d96f-4b4b-97f5-6f71025e5a85" containerName="main" Sep 16 13:49:48.373846 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.373583 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cb77e44-d96f-4b4b-97f5-6f71025e5a85" containerName="main" Sep 16 13:49:48.373846 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.373660 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="588ea072-823f-4eed-a7ef-626a3a130af5" containerName="revoke-keys" Sep 16 13:49:48.373846 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.373670 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="13ef29df-6c3a-4c73-bfcf-d8b857dcb027" containerName="payload-pre-processing" Sep 16 13:49:48.373846 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.373677 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cb77e44-d96f-4b4b-97f5-6f71025e5a85" containerName="main" Sep 16 13:49:48.373846 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.373684 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d67a8d4-e44a-4604-b653-fa7cd1b6d90e" containerName="payload-pre-processing" Sep 16 13:49:48.373846 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.373690 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="db402416-c8de-4346-83fb-5997122a7272" containerName="payload-processing" Sep 16 13:49:48.377934 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.377913 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:48.380806 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.380789 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4bb29c\"/\"e2e-conflict-b-08a980e5-kserve-self-signed-certs\"" Sep 16 13:49:48.387008 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.386983 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp"] Sep 16 13:49:48.458562 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.458524 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:48.534883 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.534846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-tmp-dir\") pod \"e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:48.534883 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.534887 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-home\") pod \"e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:48.535081 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.534913 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-dshm\") pod \"e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:48.535081 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.534946 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-model-cache\") pod \"e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:48.535081 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.535037 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdh2b\" (UniqueName: \"kubernetes.io/projected/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-kube-api-access-mdh2b\") pod \"e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:48.535081 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.535062 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-tls-certs\") pod \"e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:48.572694 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.572658 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:48.636434 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.636344 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-tmp-dir\") pod \"e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:48.636434 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.636386 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-home\") pod \"e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:48.636434 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.636410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-dshm\") pod \"e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:48.636434 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.636425 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-model-cache\") pod \"e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:48.636846 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.636493 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdh2b\" (UniqueName: \"kubernetes.io/projected/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-kube-api-access-mdh2b\") pod \"e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:48.636846 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.636515 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-tls-certs\") pod \"e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:48.636945 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.636859 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-tmp-dir\") pod \"e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:48.636945 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.636896 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-home\") pod \"e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:48.637016 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.636981 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-model-cache\") pod \"e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:48.638885 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.638854 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-dshm\") pod \"e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:48.639195 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.639176 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-tls-certs\") pod \"e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:48.650154 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.650125 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdh2b\" (UniqueName: \"kubernetes.io/projected/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-kube-api-access-mdh2b\") pod \"e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:48.688157 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.688116 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:48.962052 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:48.962026 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp"] Sep 16 13:49:48.964066 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:49:48.964038 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80f45cf1_dffa_4bf0_9553_aed1ccf1ecfc.slice/crio-219ee267f06e6b425a310a6483e69bcbef060edf999e7d03cdf964a41100d86b WatchSource:0}: Error finding container 219ee267f06e6b425a310a6483e69bcbef060edf999e7d03cdf964a41100d86b: Status 404 returned error can't find the container with id 219ee267f06e6b425a310a6483e69bcbef060edf999e7d03cdf964a41100d86b Sep 16 13:49:49.336091 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:49.336056 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:49.409982 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:49.409951 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:49.649047 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:49.648973 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" event={"ID":"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc","Type":"ContainerStarted","Data":"88097e000e39f4ac39ac1776a5ae79143aaaf1f581508a9d5a23d468d05b46cb"} Sep 16 13:49:49.649047 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:49.649014 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" event={"ID":"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc","Type":"ContainerStarted","Data":"219ee267f06e6b425a310a6483e69bcbef060edf999e7d03cdf964a41100d86b"} Sep 16 13:49:49.669151 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:49.669101 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" podStartSLOduration=1.669087545 podStartE2EDuration="1.669087545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:49:49.667734578 +0000 UTC m=+1137.768547293" watchObservedRunningTime="2026-09-16 13:49:49.669087545 +0000 UTC m=+1137.769900258" Sep 16 13:49:49.832272 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:49.832240 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:49.930669 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:49.930584 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:50.007452 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:50.007422 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:51.998114 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:51.998077 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:49:55.490697 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:55.490641 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" podUID="0bfccddd-ec33-40ed-a817-e7be2dc39964" containerName="main" containerID="cri-o://c15aa8d758d2f56d9a76068a4a139223beeebdbbade73403dbc4bdaf0c1a42fd" gracePeriod=45 Sep 16 13:49:55.683969 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:55.683936 2580 generic.go:356] "Generic (PLEG): container finished" podID="0bfccddd-ec33-40ed-a817-e7be2dc39964" containerID="c15aa8d758d2f56d9a76068a4a139223beeebdbbade73403dbc4bdaf0c1a42fd" exitCode=0 Sep 16 13:49:55.684131 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:55.683964 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" event={"ID":"0bfccddd-ec33-40ed-a817-e7be2dc39964","Type":"ContainerDied","Data":"c15aa8d758d2f56d9a76068a4a139223beeebdbbade73403dbc4bdaf0c1a42fd"} Sep 16 13:49:55.741809 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:55.741754 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:49:55.807859 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:55.807824 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-dshm\") pod \"0bfccddd-ec33-40ed-a817-e7be2dc39964\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " Sep 16 13:49:55.807859 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:55.807867 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-home\") pod \"0bfccddd-ec33-40ed-a817-e7be2dc39964\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " Sep 16 13:49:55.808119 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:55.807894 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-tmp-dir\") pod \"0bfccddd-ec33-40ed-a817-e7be2dc39964\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " Sep 16 13:49:55.808119 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:55.807912 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l87nw\" (UniqueName: \"kubernetes.io/projected/0bfccddd-ec33-40ed-a817-e7be2dc39964-kube-api-access-l87nw\") pod \"0bfccddd-ec33-40ed-a817-e7be2dc39964\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " Sep 16 13:49:55.808119 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:55.807973 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bfccddd-ec33-40ed-a817-e7be2dc39964-tls-certs\") pod \"0bfccddd-ec33-40ed-a817-e7be2dc39964\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " Sep 16 13:49:55.808119 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:55.808020 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-model-cache\") pod \"0bfccddd-ec33-40ed-a817-e7be2dc39964\" (UID: \"0bfccddd-ec33-40ed-a817-e7be2dc39964\") " Sep 16 13:49:55.808325 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:55.808189 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-home" (OuterVolumeSpecName: "home") pod "0bfccddd-ec33-40ed-a817-e7be2dc39964" (UID: "0bfccddd-ec33-40ed-a817-e7be2dc39964"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:49:55.808325 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:55.808202 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0bfccddd-ec33-40ed-a817-e7be2dc39964" (UID: "0bfccddd-ec33-40ed-a817-e7be2dc39964"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:49:55.808432 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:55.808330 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-model-cache" (OuterVolumeSpecName: "model-cache") pod "0bfccddd-ec33-40ed-a817-e7be2dc39964" (UID: "0bfccddd-ec33-40ed-a817-e7be2dc39964"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:49:55.808487 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:55.808435 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:49:55.808487 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:55.808480 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:49:55.808587 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:55.808494 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:49:55.810280 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:55.810248 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-dshm" (OuterVolumeSpecName: "dshm") pod "0bfccddd-ec33-40ed-a817-e7be2dc39964" (UID: "0bfccddd-ec33-40ed-a817-e7be2dc39964"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:49:55.810399 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:55.810332 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bfccddd-ec33-40ed-a817-e7be2dc39964-kube-api-access-l87nw" (OuterVolumeSpecName: "kube-api-access-l87nw") pod "0bfccddd-ec33-40ed-a817-e7be2dc39964" (UID: "0bfccddd-ec33-40ed-a817-e7be2dc39964"). InnerVolumeSpecName "kube-api-access-l87nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:49:55.810399 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:55.810351 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bfccddd-ec33-40ed-a817-e7be2dc39964-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0bfccddd-ec33-40ed-a817-e7be2dc39964" (UID: "0bfccddd-ec33-40ed-a817-e7be2dc39964"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:49:55.909083 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:55.909039 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bfccddd-ec33-40ed-a817-e7be2dc39964-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:49:55.909083 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:55.909076 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bfccddd-ec33-40ed-a817-e7be2dc39964-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:49:55.909083 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:55.909090 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l87nw\" (UniqueName: \"kubernetes.io/projected/0bfccddd-ec33-40ed-a817-e7be2dc39964-kube-api-access-l87nw\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:49:56.690321 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:56.690242 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" Sep 16 13:49:56.690321 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:56.690276 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr" event={"ID":"0bfccddd-ec33-40ed-a817-e7be2dc39964","Type":"ContainerDied","Data":"12787059bbba730da244fe6f4b59e2e71373dd467a42bebe6ae65189e3920524"} Sep 16 13:49:56.690786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:56.690335 2580 scope.go:117] "RemoveContainer" containerID="c15aa8d758d2f56d9a76068a4a139223beeebdbbade73403dbc4bdaf0c1a42fd" Sep 16 13:49:56.728191 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:56.728163 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr"] Sep 16 13:49:56.735624 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:56.735588 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-aca190/test-model-aca190-kserve-7cdbb86b75-bq5xr"] Sep 16 13:49:58.354953 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:58.354922 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bfccddd-ec33-40ed-a817-e7be2dc39964" path="/var/lib/kubelet/pods/0bfccddd-ec33-40ed-a817-e7be2dc39964/volumes" Sep 16 13:49:58.688688 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:58.688577 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:58.688688 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:58.688664 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:58.701687 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:58.701667 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:58.713582 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:58.713560 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:49:59.163810 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.163774 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7"] Sep 16 13:49:59.164143 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.164131 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bfccddd-ec33-40ed-a817-e7be2dc39964" containerName="main" Sep 16 13:49:59.164193 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.164144 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bfccddd-ec33-40ed-a817-e7be2dc39964" containerName="main" Sep 16 13:49:59.164251 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.164218 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bfccddd-ec33-40ed-a817-e7be2dc39964" containerName="main" Sep 16 13:49:59.168765 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.168746 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:49:59.171292 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.171272 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-2b6ce0\"/\"openshift-service-ca.crt\"" Sep 16 13:49:59.172280 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.172261 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-2b6ce0\"/\"default-dockercfg-zmmt2\"" Sep 16 13:49:59.172280 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.172276 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-2b6ce0\"/\"kube-root-ca.crt\"" Sep 16 13:49:59.172420 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.172264 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-2b6ce0\"/\"noten-model-2b6ce0-kserve-self-signed-certs\"" Sep 16 13:49:59.181221 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.176361 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7"] Sep 16 13:49:59.234325 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.234286 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-model-cache\") pod \"noten-model-2b6ce0-kserve-f6879fc9-ncsr7\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:49:59.234488 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.234340 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pchm2\" (UniqueName: \"kubernetes.io/projected/aad920f1-2db8-424d-9b3d-c1fe392b641a-kube-api-access-pchm2\") pod \"noten-model-2b6ce0-kserve-f6879fc9-ncsr7\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:49:59.234488 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.234384 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad920f1-2db8-424d-9b3d-c1fe392b641a-tls-certs\") pod \"noten-model-2b6ce0-kserve-f6879fc9-ncsr7\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:49:59.234488 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.234406 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-home\") pod \"noten-model-2b6ce0-kserve-f6879fc9-ncsr7\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:49:59.234488 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.234466 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-dshm\") pod \"noten-model-2b6ce0-kserve-f6879fc9-ncsr7\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:49:59.234662 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.234539 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-tmp-dir\") pod \"noten-model-2b6ce0-kserve-f6879fc9-ncsr7\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:49:59.335724 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.335686 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-tmp-dir\") pod \"noten-model-2b6ce0-kserve-f6879fc9-ncsr7\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:49:59.335868 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.335746 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-model-cache\") pod \"noten-model-2b6ce0-kserve-f6879fc9-ncsr7\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:49:59.335868 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.335768 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pchm2\" (UniqueName: \"kubernetes.io/projected/aad920f1-2db8-424d-9b3d-c1fe392b641a-kube-api-access-pchm2\") pod \"noten-model-2b6ce0-kserve-f6879fc9-ncsr7\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:49:59.335868 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.335797 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad920f1-2db8-424d-9b3d-c1fe392b641a-tls-certs\") pod \"noten-model-2b6ce0-kserve-f6879fc9-ncsr7\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:49:59.335868 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.335812 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-home\") pod \"noten-model-2b6ce0-kserve-f6879fc9-ncsr7\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:49:59.335868 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.335840 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-dshm\") pod \"noten-model-2b6ce0-kserve-f6879fc9-ncsr7\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:49:59.336172 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.336149 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-tmp-dir\") pod \"noten-model-2b6ce0-kserve-f6879fc9-ncsr7\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:49:59.336246 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.336179 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-model-cache\") pod \"noten-model-2b6ce0-kserve-f6879fc9-ncsr7\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:49:59.336302 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.336264 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-home\") pod \"noten-model-2b6ce0-kserve-f6879fc9-ncsr7\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:49:59.338205 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.338188 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-dshm\") pod \"noten-model-2b6ce0-kserve-f6879fc9-ncsr7\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:49:59.338455 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.338439 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad920f1-2db8-424d-9b3d-c1fe392b641a-tls-certs\") pod \"noten-model-2b6ce0-kserve-f6879fc9-ncsr7\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:49:59.345312 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.345291 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pchm2\" (UniqueName: \"kubernetes.io/projected/aad920f1-2db8-424d-9b3d-c1fe392b641a-kube-api-access-pchm2\") pod \"noten-model-2b6ce0-kserve-f6879fc9-ncsr7\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:49:59.485661 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.485600 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:49:59.570311 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.570262 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:59.763193 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:49:59.763121 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7"] Sep 16 13:49:59.766548 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:49:59.766515 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaad920f1_2db8_424d_9b3d_c1fe392b641a.slice/crio-d7a5ed969b69b095e180a58b012059760d9e88a12d2a32e9eb1b346f80acc5fe WatchSource:0}: Error finding container d7a5ed969b69b095e180a58b012059760d9e88a12d2a32e9eb1b346f80acc5fe: Status 404 returned error can't find the container with id d7a5ed969b69b095e180a58b012059760d9e88a12d2a32e9eb1b346f80acc5fe Sep 16 13:50:00.717669 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:00.717636 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" event={"ID":"aad920f1-2db8-424d-9b3d-c1fe392b641a","Type":"ContainerStarted","Data":"6e0079df8801b3df663327a92f39f535e053e013f60a9ba733d9010bf70a5f71"} Sep 16 13:50:00.718084 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:00.717681 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" event={"ID":"aad920f1-2db8-424d-9b3d-c1fe392b641a","Type":"ContainerStarted","Data":"d7a5ed969b69b095e180a58b012059760d9e88a12d2a32e9eb1b346f80acc5fe"} Sep 16 13:50:00.737052 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:00.736998 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" podStartSLOduration=1.736984338 podStartE2EDuration="1.736984338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:50:00.736892037 +0000 UTC m=+1148.837704751" watchObservedRunningTime="2026-09-16 13:50:00.736984338 +0000 UTC m=+1148.837797050" Sep 16 13:50:05.081716 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:05.081685 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn"] Sep 16 13:50:05.102419 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:05.102390 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:05.659839 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:05.659801 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:06.505538 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:06.505502 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:07.454166 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:07.454134 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:08.386450 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:08.386424 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:09.486492 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:09.486454 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:50:09.486905 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:09.486508 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:50:09.499339 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:09.499316 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:50:09.560029 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:09.559999 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:09.774339 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:09.774251 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:50:10.331031 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:10.330996 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:10.688415 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:10.688325 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:11.055108 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.055069 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:11.365287 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.365216 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8"] Sep 16 13:50:11.368780 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.368764 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.371680 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.371658 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-f2afd5-openshift-default-dockercfg-gv8cg\"" Sep 16 13:50:11.378937 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.378907 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:11.380518 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.380495 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8"] Sep 16 13:50:11.445989 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.445955 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c6c2\" (UniqueName: \"kubernetes.io/projected/5d852849-d15b-4de8-9b0c-3d4925a61534-kube-api-access-6c6c2\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.445989 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.445992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-podinfo\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.446195 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.446025 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-credential-socket\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.446195 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.446050 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-envoy\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.446195 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.446109 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-token\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.446195 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.446140 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-workload-certs\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.446195 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.446189 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-workload-socket\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.446359 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.446204 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d852849-d15b-4de8-9b0c-3d4925a61534-istiod-ca-cert\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.446359 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.446221 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-data\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.479788 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.479755 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:11.547170 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.547136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-workload-certs\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.547340 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.547189 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-workload-socket\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.547340 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.547208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d852849-d15b-4de8-9b0c-3d4925a61534-istiod-ca-cert\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.547340 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.547230 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-data\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.547340 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.547272 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c6c2\" (UniqueName: \"kubernetes.io/projected/5d852849-d15b-4de8-9b0c-3d4925a61534-kube-api-access-6c6c2\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.547340 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.547301 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-podinfo\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.547340 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.547327 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-credential-socket\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.547728 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.547356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-envoy\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.547728 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.547451 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-token\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.547728 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.547575 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-workload-socket\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.547728 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.547702 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-data\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.547917 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.547803 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-workload-certs\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.547917 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.547815 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-credential-socket\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.547917 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.547901 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d852849-d15b-4de8-9b0c-3d4925a61534-istiod-ca-cert\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.549852 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.549828 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-envoy\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.550011 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.549991 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-podinfo\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.558320 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.558258 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-token\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.558504 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.558476 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c6c2\" (UniqueName: \"kubernetes.io/projected/5d852849-d15b-4de8-9b0c-3d4925a61534-kube-api-access-6c6c2\") pod \"e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.680293 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.680192 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:11.829911 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.829868 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:11.958383 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.958358 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8"] Sep 16 13:50:11.960913 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:50:11.960887 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d852849_d15b_4de8_9b0c_3d4925a61534.slice/crio-9b97f85b7b3dbfeade2a2955f26ee235b4345103e127c8e18eb92b40f05b2086 WatchSource:0}: Error finding container 9b97f85b7b3dbfeade2a2955f26ee235b4345103e127c8e18eb92b40f05b2086: Status 404 returned error can't find the container with id 9b97f85b7b3dbfeade2a2955f26ee235b4345103e127c8e18eb92b40f05b2086 Sep 16 13:50:11.963736 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.963702 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:50:11.963822 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.963776 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:50:11.963822 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:11.963806 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:50:12.090762 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:12.090721 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:12.780764 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:12.780727 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" event={"ID":"5d852849-d15b-4de8-9b0c-3d4925a61534","Type":"ContainerStarted","Data":"0be99a08b3d3f1c82ae5edc57a015094407d848c14e0e3dd15f72559a09d51c0"} Sep 16 13:50:12.780970 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:12.780782 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" event={"ID":"5d852849-d15b-4de8-9b0c-3d4925a61534","Type":"ContainerStarted","Data":"9b97f85b7b3dbfeade2a2955f26ee235b4345103e127c8e18eb92b40f05b2086"} Sep 16 13:50:12.812099 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:12.808376 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" podStartSLOduration=1.808357963 podStartE2EDuration="1.808357963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:50:12.804283712 +0000 UTC m=+1160.905096470" watchObservedRunningTime="2026-09-16 13:50:12.808357963 +0000 UTC m=+1160.909170679" Sep 16 13:50:12.820814 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:12.820774 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:13.520996 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:13.520918 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:13.681165 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:13.681129 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:13.686829 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:13.686802 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:13.786229 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:13.786141 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:13.787201 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:13.787182 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:14.463185 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:14.463152 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:15.074165 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:15.074132 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:15.166470 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:15.166435 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:15.637807 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:15.637771 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:16.423085 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:16.423048 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:16.511488 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:16.511451 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:16.588881 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:16.588846 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:16.635845 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:16.635808 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:50:16.676988 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:16.676910 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:16.758137 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:16.758105 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:17.120121 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:17.120092 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:17.356037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:17.356007 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:18.370289 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:18.370254 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:19.133984 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:19.133948 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:19.825906 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:19.825840 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:20.100857 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.100754 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" podUID="82b09a78-25ab-4656-9261-49076035b91c" containerName="main" containerID="cri-o://9cdb7df397b882a1b7df82e8df04cfe003deaeb36d9911b658f92bc42fae4115" gracePeriod=45 Sep 16 13:50:20.358746 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.358680 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:50:20.434152 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.434119 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-model-cache\") pod \"82b09a78-25ab-4656-9261-49076035b91c\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " Sep 16 13:50:20.434325 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.434167 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-home\") pod \"82b09a78-25ab-4656-9261-49076035b91c\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " Sep 16 13:50:20.434325 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.434239 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnzqd\" (UniqueName: \"kubernetes.io/projected/82b09a78-25ab-4656-9261-49076035b91c-kube-api-access-fnzqd\") pod \"82b09a78-25ab-4656-9261-49076035b91c\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " Sep 16 13:50:20.434325 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.434273 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b09a78-25ab-4656-9261-49076035b91c-tls-certs\") pod \"82b09a78-25ab-4656-9261-49076035b91c\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " Sep 16 13:50:20.434325 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.434299 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-dshm\") pod \"82b09a78-25ab-4656-9261-49076035b91c\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " Sep 16 13:50:20.434325 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.434324 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-tmp-dir\") pod \"82b09a78-25ab-4656-9261-49076035b91c\" (UID: \"82b09a78-25ab-4656-9261-49076035b91c\") " Sep 16 13:50:20.434583 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.434440 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-home" (OuterVolumeSpecName: "home") pod "82b09a78-25ab-4656-9261-49076035b91c" (UID: "82b09a78-25ab-4656-9261-49076035b91c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:50:20.434583 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.434446 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-model-cache" (OuterVolumeSpecName: "model-cache") pod "82b09a78-25ab-4656-9261-49076035b91c" (UID: "82b09a78-25ab-4656-9261-49076035b91c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:50:20.434724 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.434669 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:20.434724 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.434690 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:20.434724 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.434713 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "82b09a78-25ab-4656-9261-49076035b91c" (UID: "82b09a78-25ab-4656-9261-49076035b91c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:50:20.436830 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.436789 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-dshm" (OuterVolumeSpecName: "dshm") pod "82b09a78-25ab-4656-9261-49076035b91c" (UID: "82b09a78-25ab-4656-9261-49076035b91c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:50:20.436946 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.436866 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82b09a78-25ab-4656-9261-49076035b91c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "82b09a78-25ab-4656-9261-49076035b91c" (UID: "82b09a78-25ab-4656-9261-49076035b91c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:50:20.436946 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.436868 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b09a78-25ab-4656-9261-49076035b91c-kube-api-access-fnzqd" (OuterVolumeSpecName: "kube-api-access-fnzqd") pod "82b09a78-25ab-4656-9261-49076035b91c" (UID: "82b09a78-25ab-4656-9261-49076035b91c"). InnerVolumeSpecName "kube-api-access-fnzqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:50:20.535678 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.535597 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b09a78-25ab-4656-9261-49076035b91c-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:20.535678 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.535672 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:20.535678 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.535682 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82b09a78-25ab-4656-9261-49076035b91c-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:20.535913 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.535691 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnzqd\" (UniqueName: \"kubernetes.io/projected/82b09a78-25ab-4656-9261-49076035b91c-kube-api-access-fnzqd\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:20.642630 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.642527 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:20.709906 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.709875 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:20.824798 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.824762 2580 generic.go:356] "Generic (PLEG): container finished" podID="82b09a78-25ab-4656-9261-49076035b91c" containerID="9cdb7df397b882a1b7df82e8df04cfe003deaeb36d9911b658f92bc42fae4115" exitCode=0 Sep 16 13:50:20.824945 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.824808 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" Sep 16 13:50:20.824945 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.824853 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" event={"ID":"82b09a78-25ab-4656-9261-49076035b91c","Type":"ContainerDied","Data":"9cdb7df397b882a1b7df82e8df04cfe003deaeb36d9911b658f92bc42fae4115"} Sep 16 13:50:20.824945 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.824896 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn" event={"ID":"82b09a78-25ab-4656-9261-49076035b91c","Type":"ContainerDied","Data":"b3772a3ce8a73864e83e378c7fd34cf809295f05dd852508db2b3f5f028d1a24"} Sep 16 13:50:20.824945 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.824914 2580 scope.go:117] "RemoveContainer" containerID="9cdb7df397b882a1b7df82e8df04cfe003deaeb36d9911b658f92bc42fae4115" Sep 16 13:50:20.834176 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.834157 2580 scope.go:117] "RemoveContainer" containerID="9cdb7df397b882a1b7df82e8df04cfe003deaeb36d9911b658f92bc42fae4115" Sep 16 13:50:20.834489 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:50:20.834448 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cdb7df397b882a1b7df82e8df04cfe003deaeb36d9911b658f92bc42fae4115\": container with ID starting with 9cdb7df397b882a1b7df82e8df04cfe003deaeb36d9911b658f92bc42fae4115 not found: ID does not exist" containerID="9cdb7df397b882a1b7df82e8df04cfe003deaeb36d9911b658f92bc42fae4115" Sep 16 13:50:20.834489 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.834473 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cdb7df397b882a1b7df82e8df04cfe003deaeb36d9911b658f92bc42fae4115"} err="failed to get container status \"9cdb7df397b882a1b7df82e8df04cfe003deaeb36d9911b658f92bc42fae4115\": rpc error: code = NotFound desc = could not find container \"9cdb7df397b882a1b7df82e8df04cfe003deaeb36d9911b658f92bc42fae4115\": container with ID starting with 9cdb7df397b882a1b7df82e8df04cfe003deaeb36d9911b658f92bc42fae4115 not found: ID does not exist" Sep 16 13:50:20.852212 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.852181 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn"] Sep 16 13:50:20.857826 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.857800 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1e5db0/test-model-1e5db0-kserve-647b747-wzvxn"] Sep 16 13:50:20.886839 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.886803 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9"] Sep 16 13:50:20.887172 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.887160 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82b09a78-25ab-4656-9261-49076035b91c" containerName="main" Sep 16 13:50:20.887217 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.887174 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="82b09a78-25ab-4656-9261-49076035b91c" containerName="main" Sep 16 13:50:20.887252 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.887245 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="82b09a78-25ab-4656-9261-49076035b91c" containerName="main" Sep 16 13:50:20.891497 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.891480 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:20.894345 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.894301 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-46ccbd\"/\"distinct-2-w0-46ccbd-kserve-self-signed-certs\"" Sep 16 13:50:20.898749 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.898724 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9"] Sep 16 13:50:20.916167 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.916139 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:20.940534 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.940499 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-home\") pod \"distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:20.940734 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.940544 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-tmp-dir\") pod \"distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:20.940734 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.940575 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-model-cache\") pod \"distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:20.940734 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.940615 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd672f1-f0ba-4787-8666-67f462e15c84-tls-certs\") pod \"distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:20.940734 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.940640 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cst99\" (UniqueName: \"kubernetes.io/projected/fbd672f1-f0ba-4787-8666-67f462e15c84-kube-api-access-cst99\") pod \"distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:20.940734 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:20.940667 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-dshm\") pod \"distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:21.041579 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:21.041542 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-home\") pod \"distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:21.041790 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:21.041586 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-tmp-dir\") pod \"distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:21.041790 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:21.041640 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-model-cache\") pod \"distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:21.041790 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:21.041677 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd672f1-f0ba-4787-8666-67f462e15c84-tls-certs\") pod \"distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:21.041790 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:21.041696 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cst99\" (UniqueName: \"kubernetes.io/projected/fbd672f1-f0ba-4787-8666-67f462e15c84-kube-api-access-cst99\") pod \"distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:21.041790 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:21.041763 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-dshm\") pod \"distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:21.042069 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:21.042020 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-home\") pod \"distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:21.042127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:21.042088 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-model-cache\") pod \"distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:21.042127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:21.042103 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-tmp-dir\") pod \"distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:21.044207 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:21.044182 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-dshm\") pod \"distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:21.044397 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:21.044379 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd672f1-f0ba-4787-8666-67f462e15c84-tls-certs\") pod \"distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:21.050307 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:21.050289 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cst99\" (UniqueName: \"kubernetes.io/projected/fbd672f1-f0ba-4787-8666-67f462e15c84-kube-api-access-cst99\") pod \"distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:21.137727 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:21.137688 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:21.201366 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:21.201276 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:21.268311 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:21.268260 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:21.486825 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:21.486796 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9"] Sep 16 13:50:21.489024 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:50:21.489000 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbd672f1_f0ba_4787_8666_67f462e15c84.slice/crio-32c99320782d70359e1b1f98793f5be67225eb4c08900585ff46af4b62767d2c WatchSource:0}: Error finding container 32c99320782d70359e1b1f98793f5be67225eb4c08900585ff46af4b62767d2c: Status 404 returned error can't find the container with id 32c99320782d70359e1b1f98793f5be67225eb4c08900585ff46af4b62767d2c Sep 16 13:50:21.672340 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:21.672305 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:21.833141 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:21.833046 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" event={"ID":"fbd672f1-f0ba-4787-8666-67f462e15c84","Type":"ContainerStarted","Data":"ec613a09259ec71c541f3e233d0f26004f5b64efc247ade5c66468fc73fd3429"} Sep 16 13:50:21.833141 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:21.833091 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" event={"ID":"fbd672f1-f0ba-4787-8666-67f462e15c84","Type":"ContainerStarted","Data":"32c99320782d70359e1b1f98793f5be67225eb4c08900585ff46af4b62767d2c"} Sep 16 13:50:21.851077 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:21.851022 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" podStartSLOduration=1.8510046839999998 podStartE2EDuration="1.851004684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:50:21.849974394 +0000 UTC m=+1169.950787111" watchObservedRunningTime="2026-09-16 13:50:21.851004684 +0000 UTC m=+1169.951817399" Sep 16 13:50:22.355471 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:22.355438 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82b09a78-25ab-4656-9261-49076035b91c" path="/var/lib/kubelet/pods/82b09a78-25ab-4656-9261-49076035b91c/volumes" Sep 16 13:50:22.545447 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:22.545414 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:23.456419 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:23.456382 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:23.685886 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:23.685858 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:23.786380 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:23.786344 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:23.961305 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:23.961271 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:24.282372 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:24.282335 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e5db0-cl6g4"] Sep 16 13:50:24.286123 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:24.286099 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e5db0-cl6g4" Sep 16 13:50:24.288737 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:24.288711 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:50:24.288864 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:24.288711 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-thk5m\"" Sep 16 13:50:24.294326 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:24.294298 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e5db0-cl6g4"] Sep 16 13:50:24.375081 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:24.375045 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f677a49-0733-4a17-803d-e7bbf6e592aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1e5db0-cl6g4\" (UID: \"6f677a49-0733-4a17-803d-e7bbf6e592aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e5db0-cl6g4" Sep 16 13:50:24.475922 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:24.475885 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f677a49-0733-4a17-803d-e7bbf6e592aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1e5db0-cl6g4\" (UID: \"6f677a49-0733-4a17-803d-e7bbf6e592aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e5db0-cl6g4" Sep 16 13:50:24.476651 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:24.476623 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f677a49-0733-4a17-803d-e7bbf6e592aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1e5db0-cl6g4\" (UID: \"6f677a49-0733-4a17-803d-e7bbf6e592aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e5db0-cl6g4" Sep 16 13:50:24.516966 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:24.516925 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:24.598360 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:24.598266 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e5db0-cl6g4" Sep 16 13:50:24.870117 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:24.870081 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e5db0-cl6g4"] Sep 16 13:50:24.872911 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:50:24.872879 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f677a49_0733_4a17_803d_e7bbf6e592aa.slice/crio-3e24b10565039c420fde4643c2024729887283a10202ff4db143848b821ce3fc WatchSource:0}: Error finding container 3e24b10565039c420fde4643c2024729887283a10202ff4db143848b821ce3fc: Status 404 returned error can't find the container with id 3e24b10565039c420fde4643c2024729887283a10202ff4db143848b821ce3fc Sep 16 13:50:25.248499 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:25.248464 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:25.662475 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:25.662446 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:25.722517 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:25.722482 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:25.855533 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:25.855443 2580 generic.go:356] "Generic (PLEG): container finished" podID="6f677a49-0733-4a17-803d-e7bbf6e592aa" containerID="c47a6f1595735cdef9f2b125768ae34afb6954b0b26015c3b4907b8c34bd6c80" exitCode=0 Sep 16 13:50:25.855726 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:25.855523 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e5db0-cl6g4" event={"ID":"6f677a49-0733-4a17-803d-e7bbf6e592aa","Type":"ContainerDied","Data":"c47a6f1595735cdef9f2b125768ae34afb6954b0b26015c3b4907b8c34bd6c80"} Sep 16 13:50:25.855726 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:25.855578 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e5db0-cl6g4" event={"ID":"6f677a49-0733-4a17-803d-e7bbf6e592aa","Type":"ContainerStarted","Data":"3e24b10565039c420fde4643c2024729887283a10202ff4db143848b821ce3fc"} Sep 16 13:50:26.056047 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:26.056006 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:26.752078 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:26.752042 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:27.002053 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:27.002030 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e5db0-cl6g4" Sep 16 13:50:27.103921 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:27.103829 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f677a49-0733-4a17-803d-e7bbf6e592aa-maas-api-service-ca\") pod \"6f677a49-0733-4a17-803d-e7bbf6e592aa\" (UID: \"6f677a49-0733-4a17-803d-e7bbf6e592aa\") " Sep 16 13:50:27.104210 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:27.104185 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f677a49-0733-4a17-803d-e7bbf6e592aa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6f677a49-0733-4a17-803d-e7bbf6e592aa" (UID: "6f677a49-0733-4a17-803d-e7bbf6e592aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:50:27.205220 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:27.205182 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f677a49-0733-4a17-803d-e7bbf6e592aa-maas-api-service-ca\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:27.519560 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:27.519526 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:27.868173 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:27.868089 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e5db0-cl6g4" Sep 16 13:50:27.868173 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:27.868128 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e5db0-cl6g4" event={"ID":"6f677a49-0733-4a17-803d-e7bbf6e592aa","Type":"ContainerDied","Data":"3e24b10565039c420fde4643c2024729887283a10202ff4db143848b821ce3fc"} Sep 16 13:50:27.868173 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:27.868166 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e24b10565039c420fde4643c2024729887283a10202ff4db143848b821ce3fc" Sep 16 13:50:28.436889 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:28.436853 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:29.173846 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:29.173805 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:31.201847 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:31.201804 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:31.201847 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:31.201857 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:31.214539 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:31.214506 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:31.901388 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:31.901351 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:50:34.582638 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:34.580247 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:34.788485 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:34.788440 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:34.925010 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:34.924926 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:34.956920 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:34.956885 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1e5db0-29826105-tl7j6"] Sep 16 13:50:34.975707 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:34.975672 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1e5db0-29826105-tl7j6"] Sep 16 13:50:34.988187 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:34.988156 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5"] Sep 16 13:50:34.988695 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:34.988656 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5" podUID="28b781ab-6240-48ef-bb22-243ba2ea1428" containerName="payload-processing" containerID="cri-o://cb6d61ced61f34ea96860bef338fd47aae2871867686e9c1da1a185a32522f8f" gracePeriod=30 Sep 16 13:50:35.015119 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.015088 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:35.085294 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.085260 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:35.159189 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.159130 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e5db0-cl6g4"] Sep 16 13:50:35.161106 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.160971 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:35.181865 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.181742 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e5db0-cl6g4"] Sep 16 13:50:35.223396 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.223359 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:35.287125 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.287085 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5" Sep 16 13:50:35.391318 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.391283 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nqdc\" (UniqueName: \"kubernetes.io/projected/28b781ab-6240-48ef-bb22-243ba2ea1428-kube-api-access-2nqdc\") pod \"28b781ab-6240-48ef-bb22-243ba2ea1428\" (UID: \"28b781ab-6240-48ef-bb22-243ba2ea1428\") " Sep 16 13:50:35.391543 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.391464 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28b781ab-6240-48ef-bb22-243ba2ea1428-plugins-config-volume\") pod \"28b781ab-6240-48ef-bb22-243ba2ea1428\" (UID: \"28b781ab-6240-48ef-bb22-243ba2ea1428\") " Sep 16 13:50:35.391779 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.391754 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b781ab-6240-48ef-bb22-243ba2ea1428-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "28b781ab-6240-48ef-bb22-243ba2ea1428" (UID: "28b781ab-6240-48ef-bb22-243ba2ea1428"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:50:35.393559 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.393529 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b781ab-6240-48ef-bb22-243ba2ea1428-kube-api-access-2nqdc" (OuterVolumeSpecName: "kube-api-access-2nqdc") pod "28b781ab-6240-48ef-bb22-243ba2ea1428" (UID: "28b781ab-6240-48ef-bb22-243ba2ea1428"). InnerVolumeSpecName "kube-api-access-2nqdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:50:35.492696 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.492658 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28b781ab-6240-48ef-bb22-243ba2ea1428-plugins-config-volume\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:35.492696 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.492689 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nqdc\" (UniqueName: \"kubernetes.io/projected/28b781ab-6240-48ef-bb22-243ba2ea1428-kube-api-access-2nqdc\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:35.524231 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.524194 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:35.624436 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.624397 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:35.920595 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.920502 2580 generic.go:356] "Generic (PLEG): container finished" podID="28b781ab-6240-48ef-bb22-243ba2ea1428" containerID="cb6d61ced61f34ea96860bef338fd47aae2871867686e9c1da1a185a32522f8f" exitCode=0 Sep 16 13:50:35.920595 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.920546 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5" Sep 16 13:50:35.920812 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.920588 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5" event={"ID":"28b781ab-6240-48ef-bb22-243ba2ea1428","Type":"ContainerDied","Data":"cb6d61ced61f34ea96860bef338fd47aae2871867686e9c1da1a185a32522f8f"} Sep 16 13:50:35.920812 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.920706 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5" event={"ID":"28b781ab-6240-48ef-bb22-243ba2ea1428","Type":"ContainerDied","Data":"7c0194c3e616727beebcd87be233465bce127ea190a3861e24daf92b395fa411"} Sep 16 13:50:35.920812 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.920730 2580 scope.go:117] "RemoveContainer" containerID="cb6d61ced61f34ea96860bef338fd47aae2871867686e9c1da1a185a32522f8f" Sep 16 13:50:35.930689 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.930672 2580 scope.go:117] "RemoveContainer" containerID="cb6d61ced61f34ea96860bef338fd47aae2871867686e9c1da1a185a32522f8f" Sep 16 13:50:35.930971 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:50:35.930953 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb6d61ced61f34ea96860bef338fd47aae2871867686e9c1da1a185a32522f8f\": container with ID starting with cb6d61ced61f34ea96860bef338fd47aae2871867686e9c1da1a185a32522f8f not found: ID does not exist" containerID="cb6d61ced61f34ea96860bef338fd47aae2871867686e9c1da1a185a32522f8f" Sep 16 13:50:35.931018 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.930980 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb6d61ced61f34ea96860bef338fd47aae2871867686e9c1da1a185a32522f8f"} err="failed to get container status \"cb6d61ced61f34ea96860bef338fd47aae2871867686e9c1da1a185a32522f8f\": rpc error: code = NotFound desc = could not find container \"cb6d61ced61f34ea96860bef338fd47aae2871867686e9c1da1a185a32522f8f\": container with ID starting with cb6d61ced61f34ea96860bef338fd47aae2871867686e9c1da1a185a32522f8f not found: ID does not exist" Sep 16 13:50:35.947501 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.947473 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5"] Sep 16 13:50:35.955315 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:35.955290 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1e5db0-697b97ff69-7zpn5"] Sep 16 13:50:36.348053 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:36.348022 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:36.354985 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:36.354950 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28b781ab-6240-48ef-bb22-243ba2ea1428" path="/var/lib/kubelet/pods/28b781ab-6240-48ef-bb22-243ba2ea1428/volumes" Sep 16 13:50:36.355370 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:36.355350 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f677a49-0733-4a17-803d-e7bbf6e592aa" path="/var/lib/kubelet/pods/6f677a49-0733-4a17-803d-e7bbf6e592aa/volumes" Sep 16 13:50:36.355699 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:36.355682 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5b1b84f-c3e2-4f76-b1ab-2d9a11e69cd2" path="/var/lib/kubelet/pods/b5b1b84f-c3e2-4f76-b1ab-2d9a11e69cd2/volumes" Sep 16 13:50:37.130918 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:37.130854 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:38.040414 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:38.040374 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:38.762531 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:38.762493 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:39.537756 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:39.537717 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:39.892935 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:39.892892 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7"] Sep 16 13:50:40.255975 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:40.255936 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:40.318365 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:40.318322 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:40.397337 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:40.397301 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:41.112622 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:41.112577 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:41.829686 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:41.829644 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:42.104503 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:42.104473 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:42.180919 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:42.180881 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:42.365128 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:42.365031 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt"] Sep 16 13:50:42.365325 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:42.365298 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" podUID="fcef5277-4be2-4ffa-8dd1-b38c60f162aa" containerName="istio-proxy" containerID="cri-o://a6b33fe23eec663ce3ce4b8f36815991855cab9920d7b83c1f97a9cf5b1bc6ae" gracePeriod=30 Sep 16 13:50:42.380354 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:42.380319 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:42.673186 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:42.673098 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:42.961902 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:42.961822 2580 generic.go:356] "Generic (PLEG): container finished" podID="5d852849-d15b-4de8-9b0c-3d4925a61534" containerID="0be99a08b3d3f1c82ae5edc57a015094407d848c14e0e3dd15f72559a09d51c0" exitCode=0 Sep 16 13:50:42.961902 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:42.961897 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" event={"ID":"5d852849-d15b-4de8-9b0c-3d4925a61534","Type":"ContainerDied","Data":"0be99a08b3d3f1c82ae5edc57a015094407d848c14e0e3dd15f72559a09d51c0"} Sep 16 13:50:42.962348 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:42.962328 2580 scope.go:117] "RemoveContainer" containerID="0be99a08b3d3f1c82ae5edc57a015094407d848c14e0e3dd15f72559a09d51c0" Sep 16 13:50:42.963441 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:42.963408 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:50:42.963558 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:42.963492 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:50:42.963558 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:42.963542 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:50:42.981668 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:42.981643 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:43.334281 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:43.334244 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:43.591513 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:43.591426 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:43.691245 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:43.691215 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:43.787073 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:43.787038 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:43.996087 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:43.996053 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" event={"ID":"5d852849-d15b-4de8-9b0c-3d4925a61534","Type":"ContainerStarted","Data":"46d94259b341e1b013a2955eecf672b2f923a75ac17f4d941f5460fea7a7ade5"} Sep 16 13:50:44.682404 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:44.682359 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:44.682973 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:44.682532 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Sep 16 13:50:44.682973 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:44.682580 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" podUID="5d852849-d15b-4de8-9b0c-3d4925a61534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Sep 16 13:50:45.492862 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:45.492806 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:45.680991 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:45.680952 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Sep 16 13:50:45.681711 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:45.681678 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" podUID="5d852849-d15b-4de8-9b0c-3d4925a61534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Sep 16 13:50:46.291546 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:46.291510 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:46.441244 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:46.441205 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:46.678228 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:46.678130 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:46.681250 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:46.681215 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Sep 16 13:50:46.681413 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:46.681275 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" podUID="5d852849-d15b-4de8-9b0c-3d4925a61534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Sep 16 13:50:47.350756 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.350723 2580 patch_prober.go:28] interesting pod/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 13:50:47.351222 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.350786 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" podUID="fcef5277-4be2-4ffa-8dd1-b38c60f162aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 13:50:47.633303 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.633277 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:50:47.689055 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.689019 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Sep 16 13:50:47.689363 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.689319 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" podUID="5d852849-d15b-4de8-9b0c-3d4925a61534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Sep 16 13:50:47.728809 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.728775 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-data\") pod \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " Sep 16 13:50:47.729012 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.728828 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-workload-socket\") pod \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " Sep 16 13:50:47.729012 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.728877 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-envoy\") pod \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " Sep 16 13:50:47.729012 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.728916 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-workload-certs\") pod \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " Sep 16 13:50:47.729012 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.728940 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntpjj\" (UniqueName: \"kubernetes.io/projected/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-kube-api-access-ntpjj\") pod \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " Sep 16 13:50:47.729012 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.728958 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-credential-socket\") pod \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " Sep 16 13:50:47.729292 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.729017 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istiod-ca-cert\") pod \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " Sep 16 13:50:47.729292 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.729069 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-podinfo\") pod \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " Sep 16 13:50:47.729292 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.729088 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-data" (OuterVolumeSpecName: "istio-data") pod "fcef5277-4be2-4ffa-8dd1-b38c60f162aa" (UID: "fcef5277-4be2-4ffa-8dd1-b38c60f162aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:50:47.729292 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.729116 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-token\") pod \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\" (UID: \"fcef5277-4be2-4ffa-8dd1-b38c60f162aa\") " Sep 16 13:50:47.729292 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.729194 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fcef5277-4be2-4ffa-8dd1-b38c60f162aa" (UID: "fcef5277-4be2-4ffa-8dd1-b38c60f162aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:50:47.729292 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.729272 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fcef5277-4be2-4ffa-8dd1-b38c60f162aa" (UID: "fcef5277-4be2-4ffa-8dd1-b38c60f162aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:50:47.729589 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.729379 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fcef5277-4be2-4ffa-8dd1-b38c60f162aa" (UID: "fcef5277-4be2-4ffa-8dd1-b38c60f162aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:50:47.729589 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.729486 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fcef5277-4be2-4ffa-8dd1-b38c60f162aa" (UID: "fcef5277-4be2-4ffa-8dd1-b38c60f162aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:50:47.729589 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.729541 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-workload-socket\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:47.729589 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.729560 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-workload-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:47.729589 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.729573 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istiod-ca-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:47.729589 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.729585 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-data\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:47.731460 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.731423 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-kube-api-access-ntpjj" (OuterVolumeSpecName: "kube-api-access-ntpjj") pod "fcef5277-4be2-4ffa-8dd1-b38c60f162aa" (UID: "fcef5277-4be2-4ffa-8dd1-b38c60f162aa"). InnerVolumeSpecName "kube-api-access-ntpjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:50:47.731566 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.731450 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-token" (OuterVolumeSpecName: "istio-token") pod "fcef5277-4be2-4ffa-8dd1-b38c60f162aa" (UID: "fcef5277-4be2-4ffa-8dd1-b38c60f162aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:50:47.732035 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.732010 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fcef5277-4be2-4ffa-8dd1-b38c60f162aa" (UID: "fcef5277-4be2-4ffa-8dd1-b38c60f162aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:50:47.732035 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.732014 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fcef5277-4be2-4ffa-8dd1-b38c60f162aa" (UID: "fcef5277-4be2-4ffa-8dd1-b38c60f162aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 13:50:47.830625 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.830581 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-envoy\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:47.830810 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.830643 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntpjj\" (UniqueName: \"kubernetes.io/projected/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-kube-api-access-ntpjj\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:47.830810 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.830659 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-credential-socket\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:47.830810 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.830670 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-podinfo\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:47.830810 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:47.830681 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcef5277-4be2-4ffa-8dd1-b38c60f162aa-istio-token\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:48.020046 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:48.020010 2580 generic.go:356] "Generic (PLEG): container finished" podID="fcef5277-4be2-4ffa-8dd1-b38c60f162aa" containerID="a6b33fe23eec663ce3ce4b8f36815991855cab9920d7b83c1f97a9cf5b1bc6ae" exitCode=0 Sep 16 13:50:48.020268 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:48.020050 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" Sep 16 13:50:48.020268 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:48.020088 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" event={"ID":"fcef5277-4be2-4ffa-8dd1-b38c60f162aa","Type":"ContainerDied","Data":"a6b33fe23eec663ce3ce4b8f36815991855cab9920d7b83c1f97a9cf5b1bc6ae"} Sep 16 13:50:48.020268 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:48.020152 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt" event={"ID":"fcef5277-4be2-4ffa-8dd1-b38c60f162aa","Type":"ContainerDied","Data":"d83721e34fdf247f4a47cabc62b16f2d209db51d66333f1ffafd84998f939e8b"} Sep 16 13:50:48.020268 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:48.020182 2580 scope.go:117] "RemoveContainer" containerID="a6b33fe23eec663ce3ce4b8f36815991855cab9920d7b83c1f97a9cf5b1bc6ae" Sep 16 13:50:48.031824 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:48.031810 2580 scope.go:117] "RemoveContainer" containerID="4d150fe896548d1e0990e83f971ef96f7ee22a37d8f761ad01bc0f2d6d713454" Sep 16 13:50:48.040212 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:48.040196 2580 scope.go:117] "RemoveContainer" containerID="a6b33fe23eec663ce3ce4b8f36815991855cab9920d7b83c1f97a9cf5b1bc6ae" Sep 16 13:50:48.040492 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:50:48.040474 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b33fe23eec663ce3ce4b8f36815991855cab9920d7b83c1f97a9cf5b1bc6ae\": container with ID starting with a6b33fe23eec663ce3ce4b8f36815991855cab9920d7b83c1f97a9cf5b1bc6ae not found: ID does not exist" containerID="a6b33fe23eec663ce3ce4b8f36815991855cab9920d7b83c1f97a9cf5b1bc6ae" Sep 16 13:50:48.040569 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:48.040505 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b33fe23eec663ce3ce4b8f36815991855cab9920d7b83c1f97a9cf5b1bc6ae"} err="failed to get container status \"a6b33fe23eec663ce3ce4b8f36815991855cab9920d7b83c1f97a9cf5b1bc6ae\": rpc error: code = NotFound desc = could not find container \"a6b33fe23eec663ce3ce4b8f36815991855cab9920d7b83c1f97a9cf5b1bc6ae\": container with ID starting with a6b33fe23eec663ce3ce4b8f36815991855cab9920d7b83c1f97a9cf5b1bc6ae not found: ID does not exist" Sep 16 13:50:48.040569 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:48.040529 2580 scope.go:117] "RemoveContainer" containerID="4d150fe896548d1e0990e83f971ef96f7ee22a37d8f761ad01bc0f2d6d713454" Sep 16 13:50:48.040841 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:50:48.040824 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d150fe896548d1e0990e83f971ef96f7ee22a37d8f761ad01bc0f2d6d713454\": container with ID starting with 4d150fe896548d1e0990e83f971ef96f7ee22a37d8f761ad01bc0f2d6d713454 not found: ID does not exist" containerID="4d150fe896548d1e0990e83f971ef96f7ee22a37d8f761ad01bc0f2d6d713454" Sep 16 13:50:48.040885 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:48.040848 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d150fe896548d1e0990e83f971ef96f7ee22a37d8f761ad01bc0f2d6d713454"} err="failed to get container status \"4d150fe896548d1e0990e83f971ef96f7ee22a37d8f761ad01bc0f2d6d713454\": rpc error: code = NotFound desc = could not find container \"4d150fe896548d1e0990e83f971ef96f7ee22a37d8f761ad01bc0f2d6d713454\": container with ID starting with 4d150fe896548d1e0990e83f971ef96f7ee22a37d8f761ad01bc0f2d6d713454 not found: ID does not exist" Sep 16 13:50:48.052498 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:48.052422 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt"] Sep 16 13:50:48.054719 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:48.054698 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1e5db0-openshift-default-5f894864b-htdrt"] Sep 16 13:50:48.073557 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:48.073495 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:48.373691 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:48.361295 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcef5277-4be2-4ffa-8dd1-b38c60f162aa" path="/var/lib/kubelet/pods/fcef5277-4be2-4ffa-8dd1-b38c60f162aa/volumes" Sep 16 13:50:48.681218 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:48.681114 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Sep 16 13:50:48.681218 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:48.681200 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" podUID="5d852849-d15b-4de8-9b0c-3d4925a61534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Sep 16 13:50:49.113947 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:49.113905 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:49.685507 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:49.685479 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:49.685953 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:49.685766 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:49.686515 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:49.686494 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:50:50.082644 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:50.082596 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:50.816250 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:50.816216 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:50:51.154707 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.154619 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:51.258487 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.258453 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:51.372730 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.372694 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:51.488488 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.488453 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:51.761661 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.761563 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:51.773558 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.773524 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs"] Sep 16 13:50:51.773926 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.773903 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcef5277-4be2-4ffa-8dd1-b38c60f162aa" containerName="istio-proxy" Sep 16 13:50:51.773926 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.773920 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcef5277-4be2-4ffa-8dd1-b38c60f162aa" containerName="istio-proxy" Sep 16 13:50:51.773926 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.773930 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28b781ab-6240-48ef-bb22-243ba2ea1428" containerName="payload-processing" Sep 16 13:50:51.774129 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.773938 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="28b781ab-6240-48ef-bb22-243ba2ea1428" containerName="payload-processing" Sep 16 13:50:51.774129 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.773956 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcef5277-4be2-4ffa-8dd1-b38c60f162aa" containerName="istio-proxy" Sep 16 13:50:51.774129 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.773964 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcef5277-4be2-4ffa-8dd1-b38c60f162aa" containerName="istio-proxy" Sep 16 13:50:51.774129 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.773990 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f677a49-0733-4a17-803d-e7bbf6e592aa" containerName="revoke-keys" Sep 16 13:50:51.774129 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.773999 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f677a49-0733-4a17-803d-e7bbf6e592aa" containerName="revoke-keys" Sep 16 13:50:51.774129 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.774087 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcef5277-4be2-4ffa-8dd1-b38c60f162aa" containerName="istio-proxy" Sep 16 13:50:51.774129 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.774099 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f677a49-0733-4a17-803d-e7bbf6e592aa" containerName="revoke-keys" Sep 16 13:50:51.774129 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.774108 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="28b781ab-6240-48ef-bb22-243ba2ea1428" containerName="payload-processing" Sep 16 13:50:51.777588 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.777562 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.782017 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.781994 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c5dc2482-openshift-default-dockercfg-rlx2b\"" Sep 16 13:50:51.792357 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.792326 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs"] Sep 16 13:50:51.850040 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.850012 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:51.872955 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.872921 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-data\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.873136 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.872972 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-credential-socket\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.873136 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.873069 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-workload-certs\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.873136 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.873112 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-envoy\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.873279 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.873145 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-workload-socket\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.873279 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.873225 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e87c6eae-3609-431f-b1cc-db13e06d7b25-istiod-ca-cert\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.873374 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.873263 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r97d9\" (UniqueName: \"kubernetes.io/projected/e87c6eae-3609-431f-b1cc-db13e06d7b25-kube-api-access-r97d9\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.873374 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.873320 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-podinfo\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.873374 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.873370 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-token\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.904174 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.904141 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v"] Sep 16 13:50:51.904586 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.904574 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcef5277-4be2-4ffa-8dd1-b38c60f162aa" containerName="istio-proxy" Sep 16 13:50:51.907515 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.907499 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v" Sep 16 13:50:51.910217 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.910194 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-973610-dockercfg-6qklp\"" Sep 16 13:50:51.910346 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.910246 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-973610\"" Sep 16 13:50:51.918288 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.918260 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v"] Sep 16 13:50:51.974755 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.974723 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-podinfo\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.974978 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.974783 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-token\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.974978 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.974961 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvbz4\" (UniqueName: \"kubernetes.io/projected/59c23cb2-d0f4-4467-a8a3-2842e3d30cf3-kube-api-access-dvbz4\") pod \"payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v\" (UID: \"59c23cb2-d0f4-4467-a8a3-2842e3d30cf3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v" Sep 16 13:50:51.975112 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.975024 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-data\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.975112 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.975044 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59c23cb2-d0f4-4467-a8a3-2842e3d30cf3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v\" (UID: \"59c23cb2-d0f4-4467-a8a3-2842e3d30cf3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v" Sep 16 13:50:51.975112 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.975094 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-credential-socket\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.975245 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.975160 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-workload-certs\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.975245 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.975198 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-envoy\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.975245 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.975225 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-workload-socket\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.975399 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.975272 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e87c6eae-3609-431f-b1cc-db13e06d7b25-istiod-ca-cert\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.975399 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.975298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r97d9\" (UniqueName: \"kubernetes.io/projected/e87c6eae-3609-431f-b1cc-db13e06d7b25-kube-api-access-r97d9\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.975503 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.975423 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-data\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.975561 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.975510 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-workload-certs\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.975670 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.975635 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-credential-socket\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.976020 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.975988 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-workload-socket\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.976020 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.976002 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e87c6eae-3609-431f-b1cc-db13e06d7b25-istiod-ca-cert\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.977682 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.977662 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-podinfo\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.978108 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.978087 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-envoy\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.996196 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.996168 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-token\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:51.996376 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:51.996352 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r97d9\" (UniqueName: \"kubernetes.io/projected/e87c6eae-3609-431f-b1cc-db13e06d7b25-kube-api-access-r97d9\") pod \"e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:52.076652 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:52.076545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvbz4\" (UniqueName: \"kubernetes.io/projected/59c23cb2-d0f4-4467-a8a3-2842e3d30cf3-kube-api-access-dvbz4\") pod \"payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v\" (UID: \"59c23cb2-d0f4-4467-a8a3-2842e3d30cf3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v" Sep 16 13:50:52.076652 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:52.076589 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59c23cb2-d0f4-4467-a8a3-2842e3d30cf3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v\" (UID: \"59c23cb2-d0f4-4467-a8a3-2842e3d30cf3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v" Sep 16 13:50:52.077265 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:52.077247 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59c23cb2-d0f4-4467-a8a3-2842e3d30cf3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v\" (UID: \"59c23cb2-d0f4-4467-a8a3-2842e3d30cf3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v" Sep 16 13:50:52.090668 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:52.090643 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvbz4\" (UniqueName: \"kubernetes.io/projected/59c23cb2-d0f4-4467-a8a3-2842e3d30cf3-kube-api-access-dvbz4\") pod \"payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v\" (UID: \"59c23cb2-d0f4-4467-a8a3-2842e3d30cf3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v" Sep 16 13:50:52.090891 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:52.090877 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:52.212487 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:52.212458 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:52.218857 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:52.218838 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v" Sep 16 13:50:52.376959 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:52.376933 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs"] Sep 16 13:50:52.379186 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:50:52.379153 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode87c6eae_3609_431f_b1cc_db13e06d7b25.slice/crio-81ad428d108354cdf3e9988e748346ec6409fdf8bbe6a2f0a68c1317e5c59d05 WatchSource:0}: Error finding container 81ad428d108354cdf3e9988e748346ec6409fdf8bbe6a2f0a68c1317e5c59d05: Status 404 returned error can't find the container with id 81ad428d108354cdf3e9988e748346ec6409fdf8bbe6a2f0a68c1317e5c59d05 Sep 16 13:50:52.381463 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:52.381427 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:50:52.381556 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:52.381491 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:50:52.381556 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:52.381518 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:50:52.476802 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:52.476772 2580 scope.go:117] "RemoveContainer" containerID="9ed29868e6f2c697cbec7efa6ebe68f146c09f118981c545159e6764c53e6fa4" Sep 16 13:50:52.502329 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:52.502296 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v"] Sep 16 13:50:52.506850 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:50:52.506812 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c23cb2_d0f4_4467_a8a3_2842e3d30cf3.slice/crio-ee25cd77db016bce4c24a49b4e42f1947109994c4ad37e270ebbe6230ca5e05f WatchSource:0}: Error finding container ee25cd77db016bce4c24a49b4e42f1947109994c4ad37e270ebbe6230ca5e05f: Status 404 returned error can't find the container with id ee25cd77db016bce4c24a49b4e42f1947109994c4ad37e270ebbe6230ca5e05f Sep 16 13:50:53.051348 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:53.051312 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" event={"ID":"e87c6eae-3609-431f-b1cc-db13e06d7b25","Type":"ContainerStarted","Data":"74af86032ffb684be380adddb7e77d3cd0fd251b26fd6e265fc3ba2fbc5209aa"} Sep 16 13:50:53.051833 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:53.051384 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" event={"ID":"e87c6eae-3609-431f-b1cc-db13e06d7b25","Type":"ContainerStarted","Data":"81ad428d108354cdf3e9988e748346ec6409fdf8bbe6a2f0a68c1317e5c59d05"} Sep 16 13:50:53.053035 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:53.053008 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v" event={"ID":"59c23cb2-d0f4-4467-a8a3-2842e3d30cf3","Type":"ContainerStarted","Data":"1b4f1092dfb4eabf00111bc29d2ede458d0e3160b44c90846d6806e2d0e53f12"} Sep 16 13:50:53.053168 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:53.053060 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v" Sep 16 13:50:53.053168 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:53.053078 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v" event={"ID":"59c23cb2-d0f4-4467-a8a3-2842e3d30cf3","Type":"ContainerStarted","Data":"ee25cd77db016bce4c24a49b4e42f1947109994c4ad37e270ebbe6230ca5e05f"} Sep 16 13:50:53.074393 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:53.074339 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" podStartSLOduration=2.074323554 podStartE2EDuration="2.074323554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:50:53.072700422 +0000 UTC m=+1201.173513137" watchObservedRunningTime="2026-09-16 13:50:53.074323554 +0000 UTC m=+1201.175136270" Sep 16 13:50:53.091284 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:53.091226 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v" podStartSLOduration=2.091210882 podStartE2EDuration="2.091210882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:50:53.089158516 +0000 UTC m=+1201.189971235" watchObservedRunningTime="2026-09-16 13:50:53.091210882 +0000 UTC m=+1201.192023597" Sep 16 13:50:53.091833 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:53.091809 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:53.097781 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:53.097755 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:53.479195 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:53.479159 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:54.059543 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:54.059514 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:54.060328 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:54.060309 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:50:54.245500 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:54.245471 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:54.913418 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:54.913376 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" podUID="aad920f1-2db8-424d-9b3d-c1fe392b641a" containerName="main" containerID="cri-o://6e0079df8801b3df663327a92f39f535e053e013f60a9ba733d9010bf70a5f71" gracePeriod=45 Sep 16 13:50:55.066176 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:55.066145 2580 generic.go:356] "Generic (PLEG): container finished" podID="aad920f1-2db8-424d-9b3d-c1fe392b641a" containerID="6e0079df8801b3df663327a92f39f535e053e013f60a9ba733d9010bf70a5f71" exitCode=0 Sep 16 13:50:55.066652 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:55.066222 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" event={"ID":"aad920f1-2db8-424d-9b3d-c1fe392b641a","Type":"ContainerDied","Data":"6e0079df8801b3df663327a92f39f535e053e013f60a9ba733d9010bf70a5f71"} Sep 16 13:50:55.163100 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:55.163069 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:50:55.184028 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:55.183945 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:55.212618 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:55.212582 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-tmp-dir\") pod \"aad920f1-2db8-424d-9b3d-c1fe392b641a\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " Sep 16 13:50:55.212808 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:55.212674 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-dshm\") pod \"aad920f1-2db8-424d-9b3d-c1fe392b641a\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " Sep 16 13:50:55.212808 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:55.212707 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pchm2\" (UniqueName: \"kubernetes.io/projected/aad920f1-2db8-424d-9b3d-c1fe392b641a-kube-api-access-pchm2\") pod \"aad920f1-2db8-424d-9b3d-c1fe392b641a\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " Sep 16 13:50:55.212808 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:55.212738 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-model-cache\") pod \"aad920f1-2db8-424d-9b3d-c1fe392b641a\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " Sep 16 13:50:55.212808 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:55.212761 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad920f1-2db8-424d-9b3d-c1fe392b641a-tls-certs\") pod \"aad920f1-2db8-424d-9b3d-c1fe392b641a\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " Sep 16 13:50:55.212808 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:55.212794 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-home\") pod \"aad920f1-2db8-424d-9b3d-c1fe392b641a\" (UID: \"aad920f1-2db8-424d-9b3d-c1fe392b641a\") " Sep 16 13:50:55.213088 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:55.212872 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aad920f1-2db8-424d-9b3d-c1fe392b641a" (UID: "aad920f1-2db8-424d-9b3d-c1fe392b641a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:50:55.213321 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:55.213174 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-home" (OuterVolumeSpecName: "home") pod "aad920f1-2db8-424d-9b3d-c1fe392b641a" (UID: "aad920f1-2db8-424d-9b3d-c1fe392b641a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:50:55.213321 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:55.213302 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-model-cache" (OuterVolumeSpecName: "model-cache") pod "aad920f1-2db8-424d-9b3d-c1fe392b641a" (UID: "aad920f1-2db8-424d-9b3d-c1fe392b641a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:50:55.213681 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:55.213396 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:55.213681 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:55.213504 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:55.213681 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:55.213647 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:55.215209 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:55.215179 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad920f1-2db8-424d-9b3d-c1fe392b641a-kube-api-access-pchm2" (OuterVolumeSpecName: "kube-api-access-pchm2") pod "aad920f1-2db8-424d-9b3d-c1fe392b641a" (UID: "aad920f1-2db8-424d-9b3d-c1fe392b641a"). InnerVolumeSpecName "kube-api-access-pchm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:50:55.215311 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:55.215222 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-dshm" (OuterVolumeSpecName: "dshm") pod "aad920f1-2db8-424d-9b3d-c1fe392b641a" (UID: "aad920f1-2db8-424d-9b3d-c1fe392b641a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:50:55.215311 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:55.215226 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aad920f1-2db8-424d-9b3d-c1fe392b641a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aad920f1-2db8-424d-9b3d-c1fe392b641a" (UID: "aad920f1-2db8-424d-9b3d-c1fe392b641a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:50:55.315131 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:55.315095 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad920f1-2db8-424d-9b3d-c1fe392b641a-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:55.315131 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:55.315124 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pchm2\" (UniqueName: \"kubernetes.io/projected/aad920f1-2db8-424d-9b3d-c1fe392b641a-kube-api-access-pchm2\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:55.315131 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:55.315135 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad920f1-2db8-424d-9b3d-c1fe392b641a-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:50:56.072915 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:56.072883 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" Sep 16 13:50:56.073368 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:56.072914 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7" event={"ID":"aad920f1-2db8-424d-9b3d-c1fe392b641a","Type":"ContainerDied","Data":"d7a5ed969b69b095e180a58b012059760d9e88a12d2a32e9eb1b346f80acc5fe"} Sep 16 13:50:56.073368 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:56.072963 2580 scope.go:117] "RemoveContainer" containerID="6e0079df8801b3df663327a92f39f535e053e013f60a9ba733d9010bf70a5f71" Sep 16 13:50:56.103956 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:56.103923 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7"] Sep 16 13:50:56.107488 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:56.107462 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-2b6ce0/noten-model-2b6ce0-kserve-f6879fc9-ncsr7"] Sep 16 13:50:56.356704 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:56.356591 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aad920f1-2db8-424d-9b3d-c1fe392b641a" path="/var/lib/kubelet/pods/aad920f1-2db8-424d-9b3d-c1fe392b641a/volumes" Sep 16 13:50:57.759110 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:57.759069 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:57.945626 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:57.945580 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:59.578367 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:50:59.578326 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:00.752795 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:00.752757 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:01.689879 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:01.689843 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:02.496386 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:02.496352 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:02.598730 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:02.598694 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:02.736556 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:02.736518 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:03.240576 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:03.240539 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:03.344527 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:03.344493 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:03.525985 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:03.525904 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l"] Sep 16 13:51:03.526452 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:03.526436 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aad920f1-2db8-424d-9b3d-c1fe392b641a" containerName="main" Sep 16 13:51:03.526503 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:03.526457 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aad920f1-2db8-424d-9b3d-c1fe392b641a" containerName="main" Sep 16 13:51:03.526570 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:03.526559 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aad920f1-2db8-424d-9b3d-c1fe392b641a" containerName="main" Sep 16 13:51:03.531665 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:03.531641 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l" Sep 16 13:51:03.539117 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:03.539093 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c5dc2482-dockercfg-8s427\"" Sep 16 13:51:03.539337 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:03.539199 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c5dc2482\"" Sep 16 13:51:03.543092 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:03.543067 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l"] Sep 16 13:51:03.699276 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:03.699237 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/955b4937-14b8-4bea-95c6-66192463e455-plugins-config-volume\") pod \"payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l\" (UID: \"955b4937-14b8-4bea-95c6-66192463e455\") " pod="openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l" Sep 16 13:51:03.699458 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:03.699316 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rblv5\" (UniqueName: \"kubernetes.io/projected/955b4937-14b8-4bea-95c6-66192463e455-kube-api-access-rblv5\") pod \"payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l\" (UID: \"955b4937-14b8-4bea-95c6-66192463e455\") " pod="openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l" Sep 16 13:51:03.800648 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:03.800531 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/955b4937-14b8-4bea-95c6-66192463e455-plugins-config-volume\") pod \"payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l\" (UID: \"955b4937-14b8-4bea-95c6-66192463e455\") " pod="openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l" Sep 16 13:51:03.800648 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:03.800624 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rblv5\" (UniqueName: \"kubernetes.io/projected/955b4937-14b8-4bea-95c6-66192463e455-kube-api-access-rblv5\") pod \"payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l\" (UID: \"955b4937-14b8-4bea-95c6-66192463e455\") " pod="openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l" Sep 16 13:51:03.801210 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:03.801190 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/955b4937-14b8-4bea-95c6-66192463e455-plugins-config-volume\") pod \"payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l\" (UID: \"955b4937-14b8-4bea-95c6-66192463e455\") " pod="openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l" Sep 16 13:51:03.813708 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:03.813674 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rblv5\" (UniqueName: \"kubernetes.io/projected/955b4937-14b8-4bea-95c6-66192463e455-kube-api-access-rblv5\") pod \"payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l\" (UID: \"955b4937-14b8-4bea-95c6-66192463e455\") " pod="openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l" Sep 16 13:51:03.843586 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:03.843559 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l" Sep 16 13:51:04.061730 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:04.061644 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v" Sep 16 13:51:04.132425 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:04.132400 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l"] Sep 16 13:51:04.135134 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:51:04.135104 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955b4937_14b8_4bea_95c6_66192463e455.slice/crio-f5c3e687d660d60c715f07df4c000aef281d77e8d46beb814f21652b8571743d WatchSource:0}: Error finding container f5c3e687d660d60c715f07df4c000aef281d77e8d46beb814f21652b8571743d: Status 404 returned error can't find the container with id f5c3e687d660d60c715f07df4c000aef281d77e8d46beb814f21652b8571743d Sep 16 13:51:05.122802 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:05.122761 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l" event={"ID":"955b4937-14b8-4bea-95c6-66192463e455","Type":"ContainerStarted","Data":"8fb8b2f6e60a47fad14c04ee3047183adc05e9fa13c8342ebfa2a5c046dcffe8"} Sep 16 13:51:05.123237 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:05.122848 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l" Sep 16 13:51:05.123237 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:05.122865 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l" event={"ID":"955b4937-14b8-4bea-95c6-66192463e455","Type":"ContainerStarted","Data":"f5c3e687d660d60c715f07df4c000aef281d77e8d46beb814f21652b8571743d"} Sep 16 13:51:13.018317 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:13.018277 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:13.118725 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:13.118685 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:14.188596 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:14.188559 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:14.985488 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:14.985452 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:16.066505 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:16.066467 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:16.130706 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:16.130678 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l" Sep 16 13:51:16.156960 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:16.156911 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l" podStartSLOduration=13.1568941 podStartE2EDuration="13.1568941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:51:05.149540471 +0000 UTC m=+1213.250353186" watchObservedRunningTime="2026-09-16 13:51:16.1568941 +0000 UTC m=+1224.257706815" Sep 16 13:51:16.285085 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:16.285047 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:16.617257 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:16.617226 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:16.910698 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:16.910599 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9"] Sep 16 13:51:16.914226 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:16.914203 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:16.921700 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:16.921672 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-8c36a6-openshift-default-dockercfg-2lvpd\"" Sep 16 13:51:16.933220 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:16.933196 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-workload-socket\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:16.933358 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:16.933241 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-token\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:16.933358 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:16.933276 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-podinfo\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:16.933358 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:16.933316 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2698b1b4-3481-41fa-8afc-79551dee1bc8-istiod-ca-cert\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:16.933358 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:16.933349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-credential-socket\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:16.933574 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:16.933371 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgrng\" (UniqueName: \"kubernetes.io/projected/2698b1b4-3481-41fa-8afc-79551dee1bc8-kube-api-access-rgrng\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:16.933574 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:16.933395 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-data\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:16.933574 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:16.933435 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-workload-certs\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:16.933574 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:16.933507 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-envoy\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:16.933574 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:16.933542 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9"] Sep 16 13:51:16.954784 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:16.954759 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:17.034476 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.034438 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2698b1b4-3481-41fa-8afc-79551dee1bc8-istiod-ca-cert\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:17.034724 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.034508 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-credential-socket\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:17.034724 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.034541 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgrng\" (UniqueName: \"kubernetes.io/projected/2698b1b4-3481-41fa-8afc-79551dee1bc8-kube-api-access-rgrng\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:17.034724 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.034565 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-data\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:17.034724 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.034643 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-workload-certs\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:17.034724 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.034674 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-envoy\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:17.035027 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.034775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-workload-socket\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:17.035027 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.034833 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-token\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:17.035027 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.034868 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-podinfo\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:17.035188 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.035020 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-credential-socket\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:17.035188 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.035095 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-data\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:17.035294 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.035186 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-workload-socket\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:17.035531 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.035501 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-workload-certs\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:17.035969 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.035945 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2698b1b4-3481-41fa-8afc-79551dee1bc8-istiod-ca-cert\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:17.037385 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.037357 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-envoy\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:17.037565 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.037548 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-podinfo\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:17.047277 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.047249 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgrng\" (UniqueName: \"kubernetes.io/projected/2698b1b4-3481-41fa-8afc-79551dee1bc8-kube-api-access-rgrng\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:17.065351 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.065317 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-token\") pod \"e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:17.143299 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.143261 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:17.225140 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.225109 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:17.310796 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.310758 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:17.541935 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.541909 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9"] Sep 16 13:51:17.544350 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:51:17.544319 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2698b1b4_3481_41fa_8afc_79551dee1bc8.slice/crio-a70278d6c7ac9afba03731d62439bcd136c7937fe220ed053346a6dd49609bf7 WatchSource:0}: Error finding container a70278d6c7ac9afba03731d62439bcd136c7937fe220ed053346a6dd49609bf7: Status 404 returned error can't find the container with id a70278d6c7ac9afba03731d62439bcd136c7937fe220ed053346a6dd49609bf7 Sep 16 13:51:17.546868 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.546830 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:51:17.546953 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.546914 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:51:17.546995 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:17.546960 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:51:18.206246 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:18.206214 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" event={"ID":"2698b1b4-3481-41fa-8afc-79551dee1bc8","Type":"ContainerStarted","Data":"cbfb5fc428aee7bccc6c3643b2edd0d534e9764259a15ee95049ce6e627a5216"} Sep 16 13:51:18.206740 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:18.206267 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" event={"ID":"2698b1b4-3481-41fa-8afc-79551dee1bc8","Type":"ContainerStarted","Data":"a70278d6c7ac9afba03731d62439bcd136c7937fe220ed053346a6dd49609bf7"} Sep 16 13:51:18.225342 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:18.225313 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:18.230713 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:18.230689 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:18.244943 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:18.244898 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" podStartSLOduration=2.24488141 podStartE2EDuration="2.24488141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:51:18.24471292 +0000 UTC m=+1226.345525638" watchObservedRunningTime="2026-09-16 13:51:18.24488141 +0000 UTC m=+1226.345694124" Sep 16 13:51:18.745998 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:18.745961 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:19.184260 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:19.184173 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:19.212192 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:19.212156 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:19.213206 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:19.213189 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:19.975791 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:19.975761 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:20.219092 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:20.219058 2580 generic.go:356] "Generic (PLEG): container finished" podID="e87c6eae-3609-431f-b1cc-db13e06d7b25" containerID="74af86032ffb684be380adddb7e77d3cd0fd251b26fd6e265fc3ba2fbc5209aa" exitCode=0 Sep 16 13:51:20.219527 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:20.219142 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" event={"ID":"e87c6eae-3609-431f-b1cc-db13e06d7b25","Type":"ContainerDied","Data":"74af86032ffb684be380adddb7e77d3cd0fd251b26fd6e265fc3ba2fbc5209aa"} Sep 16 13:51:20.219756 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:20.219738 2580 scope.go:117] "RemoveContainer" containerID="74af86032ffb684be380adddb7e77d3cd0fd251b26fd6e265fc3ba2fbc5209aa" Sep 16 13:51:20.220985 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:20.220959 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:51:20.221050 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:20.221023 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:51:20.221092 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:20.221070 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:51:20.286315 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:20.286290 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:20.453068 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:20.453015 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:21.226919 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:21.226886 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" event={"ID":"e87c6eae-3609-431f-b1cc-db13e06d7b25","Type":"ContainerStarted","Data":"8af235dcf9ec1fdf94cd0b61af7e3ccb76004abcd9d0922d6f6283010634a8c4"} Sep 16 13:51:21.340585 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:21.340549 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:22.091991 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:22.091953 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:51:22.093138 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:22.093112 2580 patch_prober.go:28] interesting pod/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 16 13:51:22.093258 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:22.093163 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" podUID="e87c6eae-3609-431f-b1cc-db13e06d7b25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 16 13:51:22.281653 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:22.281597 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:22.376442 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:22.376342 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:22.564720 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:22.564682 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:22.808549 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:22.808514 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:22.908685 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:22.908648 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:23.092094 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:23.092007 2580 patch_prober.go:28] interesting pod/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 16 13:51:23.092094 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:23.092078 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" podUID="e87c6eae-3609-431f-b1cc-db13e06d7b25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 16 13:51:23.428192 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:23.428081 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:24.059750 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:24.059718 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:51:24.091841 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:24.091805 2580 patch_prober.go:28] interesting pod/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 16 13:51:24.092011 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:24.091864 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" podUID="e87c6eae-3609-431f-b1cc-db13e06d7b25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 16 13:51:24.194037 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:24.194001 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:25.091502 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:25.091460 2580 patch_prober.go:28] interesting pod/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 16 13:51:25.091978 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:25.091522 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" podUID="e87c6eae-3609-431f-b1cc-db13e06d7b25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 16 13:51:25.617666 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:25.617623 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:25.720058 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:25.720009 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:26.092379 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:26.092339 2580 patch_prober.go:28] interesting pod/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 16 13:51:26.092860 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:26.092411 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" podUID="e87c6eae-3609-431f-b1cc-db13e06d7b25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 16 13:51:26.622677 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:26.622637 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:26.939024 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:26.938932 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:27.095331 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:27.095299 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:51:27.096374 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:27.096355 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:51:27.858093 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:27.858061 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:28.044468 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:28.044432 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp"] Sep 16 13:51:28.077839 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:28.077807 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:28.942313 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:28.942278 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:29.427974 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:29.427943 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx"] Sep 16 13:51:29.485762 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:29.485730 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:29.548142 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:29.548102 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:51:30.043171 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:30.043135 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:31.625652 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:31.625591 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:32.508751 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:32.508711 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:33.266567 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:33.266533 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:34.129872 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:34.129840 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:34.943373 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:34.943336 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:35.040719 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:35.040690 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:35.858294 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:35.858261 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:36.753931 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:36.753900 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:37.584868 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:37.584827 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:43.063982 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.063913 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" podUID="80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc" containerName="main" containerID="cri-o://88097e000e39f4ac39ac1776a5ae79143aaaf1f581508a9d5a23d468d05b46cb" gracePeriod=45 Sep 16 13:51:43.319731 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.319674 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:51:43.349110 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.349073 2580 generic.go:356] "Generic (PLEG): container finished" podID="80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc" containerID="88097e000e39f4ac39ac1776a5ae79143aaaf1f581508a9d5a23d468d05b46cb" exitCode=0 Sep 16 13:51:43.349110 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.349109 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" Sep 16 13:51:43.349336 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.349164 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" event={"ID":"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc","Type":"ContainerDied","Data":"88097e000e39f4ac39ac1776a5ae79143aaaf1f581508a9d5a23d468d05b46cb"} Sep 16 13:51:43.349336 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.349221 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp" event={"ID":"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc","Type":"ContainerDied","Data":"219ee267f06e6b425a310a6483e69bcbef060edf999e7d03cdf964a41100d86b"} Sep 16 13:51:43.349336 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.349243 2580 scope.go:117] "RemoveContainer" containerID="88097e000e39f4ac39ac1776a5ae79143aaaf1f581508a9d5a23d468d05b46cb" Sep 16 13:51:43.359495 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.359474 2580 scope.go:117] "RemoveContainer" containerID="88097e000e39f4ac39ac1776a5ae79143aaaf1f581508a9d5a23d468d05b46cb" Sep 16 13:51:43.359825 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:51:43.359804 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88097e000e39f4ac39ac1776a5ae79143aaaf1f581508a9d5a23d468d05b46cb\": container with ID starting with 88097e000e39f4ac39ac1776a5ae79143aaaf1f581508a9d5a23d468d05b46cb not found: ID does not exist" containerID="88097e000e39f4ac39ac1776a5ae79143aaaf1f581508a9d5a23d468d05b46cb" Sep 16 13:51:43.359907 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.359836 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88097e000e39f4ac39ac1776a5ae79143aaaf1f581508a9d5a23d468d05b46cb"} err="failed to get container status \"88097e000e39f4ac39ac1776a5ae79143aaaf1f581508a9d5a23d468d05b46cb\": rpc error: code = NotFound desc = could not find container \"88097e000e39f4ac39ac1776a5ae79143aaaf1f581508a9d5a23d468d05b46cb\": container with ID starting with 88097e000e39f4ac39ac1776a5ae79143aaaf1f581508a9d5a23d468d05b46cb not found: ID does not exist" Sep 16 13:51:43.397286 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.397259 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdh2b\" (UniqueName: \"kubernetes.io/projected/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-kube-api-access-mdh2b\") pod \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " Sep 16 13:51:43.397286 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.397291 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-model-cache\") pod \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " Sep 16 13:51:43.397484 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.397312 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-tmp-dir\") pod \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " Sep 16 13:51:43.397484 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.397363 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-home\") pod \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " Sep 16 13:51:43.397484 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.397379 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-tls-certs\") pod \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " Sep 16 13:51:43.397484 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.397452 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-dshm\") pod \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\" (UID: \"80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc\") " Sep 16 13:51:43.397710 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.397547 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-model-cache" (OuterVolumeSpecName: "model-cache") pod "80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc" (UID: "80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:51:43.397710 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.397656 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-home" (OuterVolumeSpecName: "home") pod "80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc" (UID: "80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:51:43.397710 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.397692 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc" (UID: "80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:51:43.397836 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.397812 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:51:43.397836 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.397825 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:51:43.397836 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.397835 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:51:43.399644 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.399593 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-kube-api-access-mdh2b" (OuterVolumeSpecName: "kube-api-access-mdh2b") pod "80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc" (UID: "80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc"). InnerVolumeSpecName "kube-api-access-mdh2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:51:43.399764 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.399689 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc" (UID: "80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:51:43.399815 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.399764 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-dshm" (OuterVolumeSpecName: "dshm") pod "80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc" (UID: "80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:51:43.498625 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.498566 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdh2b\" (UniqueName: \"kubernetes.io/projected/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-kube-api-access-mdh2b\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:51:43.498625 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.498601 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:51:43.498873 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.498647 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:51:43.680381 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.680342 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp"] Sep 16 13:51:43.682284 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:43.682256 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-b-08a980e5-kserve-594b59557f-x64qp"] Sep 16 13:51:44.363446 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:44.363411 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc" path="/var/lib/kubelet/pods/80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc/volumes" Sep 16 13:51:44.447123 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:44.447081 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" podUID="a3bde040-bc64-44a8-98b7-9abe14f6215c" containerName="main" containerID="cri-o://98687ff24ee533463b1f74e608c92849586653efb50ce052e3dfe5ecd730855a" gracePeriod=45 Sep 16 13:51:44.721915 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:44.721885 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:51:44.811233 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:44.811200 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-dshm\") pod \"a3bde040-bc64-44a8-98b7-9abe14f6215c\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " Sep 16 13:51:44.811423 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:44.811243 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-model-cache\") pod \"a3bde040-bc64-44a8-98b7-9abe14f6215c\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " Sep 16 13:51:44.811423 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:44.811301 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-tmp-dir\") pod \"a3bde040-bc64-44a8-98b7-9abe14f6215c\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " Sep 16 13:51:44.811423 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:44.811328 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdv7w\" (UniqueName: \"kubernetes.io/projected/a3bde040-bc64-44a8-98b7-9abe14f6215c-kube-api-access-jdv7w\") pod \"a3bde040-bc64-44a8-98b7-9abe14f6215c\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " Sep 16 13:51:44.811423 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:44.811369 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3bde040-bc64-44a8-98b7-9abe14f6215c-tls-certs\") pod \"a3bde040-bc64-44a8-98b7-9abe14f6215c\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " Sep 16 13:51:44.811423 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:44.811392 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-home\") pod \"a3bde040-bc64-44a8-98b7-9abe14f6215c\" (UID: \"a3bde040-bc64-44a8-98b7-9abe14f6215c\") " Sep 16 13:51:44.811768 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:44.811540 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3bde040-bc64-44a8-98b7-9abe14f6215c" (UID: "a3bde040-bc64-44a8-98b7-9abe14f6215c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:51:44.811768 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:44.811551 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3bde040-bc64-44a8-98b7-9abe14f6215c" (UID: "a3bde040-bc64-44a8-98b7-9abe14f6215c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:51:44.811768 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:44.811749 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-home" (OuterVolumeSpecName: "home") pod "a3bde040-bc64-44a8-98b7-9abe14f6215c" (UID: "a3bde040-bc64-44a8-98b7-9abe14f6215c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:51:44.811892 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:44.811758 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:51:44.811892 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:44.811793 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:51:44.813573 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:44.813545 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-dshm" (OuterVolumeSpecName: "dshm") pod "a3bde040-bc64-44a8-98b7-9abe14f6215c" (UID: "a3bde040-bc64-44a8-98b7-9abe14f6215c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:51:44.813732 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:44.813686 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3bde040-bc64-44a8-98b7-9abe14f6215c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3bde040-bc64-44a8-98b7-9abe14f6215c" (UID: "a3bde040-bc64-44a8-98b7-9abe14f6215c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:51:44.813732 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:44.813707 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3bde040-bc64-44a8-98b7-9abe14f6215c-kube-api-access-jdv7w" (OuterVolumeSpecName: "kube-api-access-jdv7w") pod "a3bde040-bc64-44a8-98b7-9abe14f6215c" (UID: "a3bde040-bc64-44a8-98b7-9abe14f6215c"). InnerVolumeSpecName "kube-api-access-jdv7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:51:44.912917 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:44.912830 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3bde040-bc64-44a8-98b7-9abe14f6215c-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:51:44.912917 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:44.912860 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:51:44.912917 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:44.912868 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3bde040-bc64-44a8-98b7-9abe14f6215c-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:51:44.912917 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:44.912877 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdv7w\" (UniqueName: \"kubernetes.io/projected/a3bde040-bc64-44a8-98b7-9abe14f6215c-kube-api-access-jdv7w\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:51:45.374057 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:45.373894 2580 generic.go:356] "Generic (PLEG): container finished" podID="a3bde040-bc64-44a8-98b7-9abe14f6215c" containerID="98687ff24ee533463b1f74e608c92849586653efb50ce052e3dfe5ecd730855a" exitCode=0 Sep 16 13:51:45.374057 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:45.373968 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" event={"ID":"a3bde040-bc64-44a8-98b7-9abe14f6215c","Type":"ContainerDied","Data":"98687ff24ee533463b1f74e608c92849586653efb50ce052e3dfe5ecd730855a"} Sep 16 13:51:45.374057 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:45.374021 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" event={"ID":"a3bde040-bc64-44a8-98b7-9abe14f6215c","Type":"ContainerDied","Data":"90d2ae653cd4a2fb382d40728658d3a2c4096b1514277901fdf8b8e5739e9a39"} Sep 16 13:51:45.374057 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:45.374044 2580 scope.go:117] "RemoveContainer" containerID="98687ff24ee533463b1f74e608c92849586653efb50ce052e3dfe5ecd730855a" Sep 16 13:51:45.380649 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:45.374734 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx" Sep 16 13:51:45.393156 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:45.393136 2580 scope.go:117] "RemoveContainer" containerID="98687ff24ee533463b1f74e608c92849586653efb50ce052e3dfe5ecd730855a" Sep 16 13:51:45.393469 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:51:45.393447 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98687ff24ee533463b1f74e608c92849586653efb50ce052e3dfe5ecd730855a\": container with ID starting with 98687ff24ee533463b1f74e608c92849586653efb50ce052e3dfe5ecd730855a not found: ID does not exist" containerID="98687ff24ee533463b1f74e608c92849586653efb50ce052e3dfe5ecd730855a" Sep 16 13:51:45.393546 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:45.393479 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98687ff24ee533463b1f74e608c92849586653efb50ce052e3dfe5ecd730855a"} err="failed to get container status \"98687ff24ee533463b1f74e608c92849586653efb50ce052e3dfe5ecd730855a\": rpc error: code = NotFound desc = could not find container \"98687ff24ee533463b1f74e608c92849586653efb50ce052e3dfe5ecd730855a\": container with ID starting with 98687ff24ee533463b1f74e608c92849586653efb50ce052e3dfe5ecd730855a not found: ID does not exist" Sep 16 13:51:45.418275 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:45.418242 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx"] Sep 16 13:51:45.421524 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:45.421501 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/e2e-conflict-a-08a980e5-kserve-585dfb86d5-5pzdx"] Sep 16 13:51:46.356197 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:46.356160 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3bde040-bc64-44a8-98b7-9abe14f6215c" path="/var/lib/kubelet/pods/a3bde040-bc64-44a8-98b7-9abe14f6215c/volumes" Sep 16 13:51:47.405104 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.405071 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2"] Sep 16 13:51:47.405515 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.405428 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3bde040-bc64-44a8-98b7-9abe14f6215c" containerName="main" Sep 16 13:51:47.405515 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.405440 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3bde040-bc64-44a8-98b7-9abe14f6215c" containerName="main" Sep 16 13:51:47.405515 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.405460 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc" containerName="main" Sep 16 13:51:47.405515 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.405465 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc" containerName="main" Sep 16 13:51:47.405758 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.405522 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3bde040-bc64-44a8-98b7-9abe14f6215c" containerName="main" Sep 16 13:51:47.405758 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.405530 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="80f45cf1-dffa-4bf0-9553-aed1ccf1ecfc" containerName="main" Sep 16 13:51:47.409887 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.409869 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:47.416206 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.416178 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c5dc2482\"/\"default-dockercfg-zmkgb\"" Sep 16 13:51:47.417199 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.417180 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c5dc2482\"/\"kube-root-ca.crt\"" Sep 16 13:51:47.417295 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.417180 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c5dc2482\"/\"e2e-lifecycle-model-c5dc2482-kserve-self-signed-certs\"" Sep 16 13:51:47.418289 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.418272 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c5dc2482\"/\"openshift-service-ca.crt\"" Sep 16 13:51:47.426704 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.426665 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2"] Sep 16 13:51:47.536473 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.536433 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13622db8-6ad3-4b22-8bde-beaf54327f81-tls-certs\") pod \"e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:47.536708 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.536548 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-tmp-dir\") pod \"e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:47.536708 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.536644 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-model-cache\") pod \"e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:47.536708 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.536673 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-dshm\") pod \"e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:47.536708 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.536697 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-home\") pod \"e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:47.536960 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.536776 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdv5n\" (UniqueName: \"kubernetes.io/projected/13622db8-6ad3-4b22-8bde-beaf54327f81-kube-api-access-kdv5n\") pod \"e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:47.582502 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.582371 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:47.638329 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.638293 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdv5n\" (UniqueName: \"kubernetes.io/projected/13622db8-6ad3-4b22-8bde-beaf54327f81-kube-api-access-kdv5n\") pod \"e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:47.638545 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.638362 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13622db8-6ad3-4b22-8bde-beaf54327f81-tls-certs\") pod \"e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:47.638545 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.638405 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-tmp-dir\") pod \"e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:47.638545 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.638433 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-model-cache\") pod \"e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:47.638545 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.638454 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-dshm\") pod \"e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:47.638545 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.638476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-home\") pod \"e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:47.639007 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.638977 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-home\") pod \"e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:47.639183 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.639016 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-tmp-dir\") pod \"e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:47.639183 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.638982 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-model-cache\") pod \"e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:47.640938 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.640909 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-dshm\") pod \"e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:47.641242 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.641221 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13622db8-6ad3-4b22-8bde-beaf54327f81-tls-certs\") pod \"e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:47.647336 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.647313 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdv5n\" (UniqueName: \"kubernetes.io/projected/13622db8-6ad3-4b22-8bde-beaf54327f81-kube-api-access-kdv5n\") pod \"e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:47.719969 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:47.719938 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:48.022732 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:48.022704 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2"] Sep 16 13:51:48.025455 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:51:48.025425 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13622db8_6ad3_4b22_8bde_beaf54327f81.slice/crio-c2e577cfd2a341e551824a1adc214580ce115db2e4d5784261d3a5e40d702948 WatchSource:0}: Error finding container c2e577cfd2a341e551824a1adc214580ce115db2e4d5784261d3a5e40d702948: Status 404 returned error can't find the container with id c2e577cfd2a341e551824a1adc214580ce115db2e4d5784261d3a5e40d702948 Sep 16 13:51:48.396628 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:48.396515 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" event={"ID":"13622db8-6ad3-4b22-8bde-beaf54327f81","Type":"ContainerStarted","Data":"c637b124f7470d4dd428dfca41c4a7ea5a93dceba0f0b69b6ec68267b6086186"} Sep 16 13:51:48.396628 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:48.396562 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" event={"ID":"13622db8-6ad3-4b22-8bde-beaf54327f81","Type":"ContainerStarted","Data":"c2e577cfd2a341e551824a1adc214580ce115db2e4d5784261d3a5e40d702948"} Sep 16 13:51:48.421544 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:48.421497 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" podStartSLOduration=1.42147938 podStartE2EDuration="1.42147938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:51:48.418157811 +0000 UTC m=+1256.518970560" watchObservedRunningTime="2026-09-16 13:51:48.42147938 +0000 UTC m=+1256.522292086" Sep 16 13:51:52.522826 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:52.522786 2580 scope.go:117] "RemoveContainer" containerID="9394956c7cca4b97a874ba2764a3e0b6f8e68d98499483f3f912273fb6a1b6fb" Sep 16 13:51:52.533516 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:52.533497 2580 scope.go:117] "RemoveContainer" containerID="284a45f0641be5861ed1779a72feab41a4e1180a47ea54261a3179accc9b64fd" Sep 16 13:51:52.541460 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:52.541442 2580 scope.go:117] "RemoveContainer" containerID="d488d9adf220157b0c83e0a457754988bbe7b68fa3c5f7408a486884d250b6c5" Sep 16 13:51:52.549364 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:52.549339 2580 scope.go:117] "RemoveContainer" containerID="e0652f3ba18fb719c9704c64fb6d69e0ef34d2d1be314f5a5158a116a6ba44b7" Sep 16 13:51:53.428498 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:53.428465 2580 generic.go:356] "Generic (PLEG): container finished" podID="2698b1b4-3481-41fa-8afc-79551dee1bc8" containerID="cbfb5fc428aee7bccc6c3643b2edd0d534e9764259a15ee95049ce6e627a5216" exitCode=0 Sep 16 13:51:53.428719 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:53.428532 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" event={"ID":"2698b1b4-3481-41fa-8afc-79551dee1bc8","Type":"ContainerDied","Data":"cbfb5fc428aee7bccc6c3643b2edd0d534e9764259a15ee95049ce6e627a5216"} Sep 16 13:51:53.429015 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:53.428990 2580 scope.go:117] "RemoveContainer" containerID="cbfb5fc428aee7bccc6c3643b2edd0d534e9764259a15ee95049ce6e627a5216" Sep 16 13:51:53.430167 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:53.430135 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:51:53.430252 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:53.430205 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:51:53.430252 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:53.430238 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:51:54.435936 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:54.435905 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" event={"ID":"2698b1b4-3481-41fa-8afc-79551dee1bc8","Type":"ContainerStarted","Data":"7af7bc12b61dd0419da45081202cc0e18088f442ee08c1ba5e97a479d395c215"} Sep 16 13:51:55.225340 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.225298 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:51:55.225557 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.225537 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Sep 16 13:51:55.225639 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.225586 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" podUID="2698b1b4-3481-41fa-8afc-79551dee1bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Sep 16 13:51:55.383832 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.383787 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt"] Sep 16 13:51:55.387108 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.387084 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:51:55.389839 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.389812 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4bb29c\"/\"distinct-w4-4bb29c-kserve-self-signed-certs\"" Sep 16 13:51:55.396811 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.396787 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt"] Sep 16 13:51:55.491976 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.491897 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:55.525404 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.525371 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-home\") pod \"distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:51:55.525559 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.525419 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-dshm\") pod \"distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:51:55.525559 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.525448 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-model-cache\") pod \"distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:51:55.525559 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.525518 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e443f63-4448-4555-8e7f-7994b79dc069-tls-certs\") pod \"distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:51:55.525559 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.525546 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4b9q\" (UniqueName: \"kubernetes.io/projected/1e443f63-4448-4555-8e7f-7994b79dc069-kube-api-access-d4b9q\") pod \"distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:51:55.525753 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.525596 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-tmp-dir\") pod \"distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:51:55.626347 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.626317 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-dshm\") pod \"distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:51:55.626547 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.626371 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-model-cache\") pod \"distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:51:55.626547 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.626411 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e443f63-4448-4555-8e7f-7994b79dc069-tls-certs\") pod \"distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:51:55.626697 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.626539 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4b9q\" (UniqueName: \"kubernetes.io/projected/1e443f63-4448-4555-8e7f-7994b79dc069-kube-api-access-d4b9q\") pod \"distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:51:55.626754 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.626704 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-tmp-dir\") pod \"distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:51:55.626957 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.626831 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-home\") pod \"distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:51:55.626957 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.626841 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-model-cache\") pod \"distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:51:55.627167 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.627075 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-tmp-dir\") pod \"distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:51:55.627167 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.627124 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-home\") pod \"distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:51:55.629007 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.628982 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-dshm\") pod \"distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:51:55.629139 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.629125 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e443f63-4448-4555-8e7f-7994b79dc069-tls-certs\") pod \"distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:51:55.636300 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.636280 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4b9q\" (UniqueName: \"kubernetes.io/projected/1e443f63-4448-4555-8e7f-7994b79dc069-kube-api-access-d4b9q\") pod \"distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:51:55.698378 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.698338 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:51:55.974348 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:55.974299 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt"] Sep 16 13:51:55.979933 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:51:55.979880 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e443f63_4448_4555_8e7f_7994b79dc069.slice/crio-71b62219f21d60c1f21036c0cf891c6d0facda2531b952b3243cf6b956f1e11e WatchSource:0}: Error finding container 71b62219f21d60c1f21036c0cf891c6d0facda2531b952b3243cf6b956f1e11e: Status 404 returned error can't find the container with id 71b62219f21d60c1f21036c0cf891c6d0facda2531b952b3243cf6b956f1e11e Sep 16 13:51:56.226518 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:56.226482 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Sep 16 13:51:56.226726 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:56.226540 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" podUID="2698b1b4-3481-41fa-8afc-79551dee1bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Sep 16 13:51:56.447198 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:56.447113 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" event={"ID":"1e443f63-4448-4555-8e7f-7994b79dc069","Type":"ContainerStarted","Data":"d4f8d9d61d30783dc881d4641514123d24c26cb0d2274c8b5f12bc1b6b1b86d0"} Sep 16 13:51:56.447198 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:56.447154 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" event={"ID":"1e443f63-4448-4555-8e7f-7994b79dc069","Type":"ContainerStarted","Data":"71b62219f21d60c1f21036c0cf891c6d0facda2531b952b3243cf6b956f1e11e"} Sep 16 13:51:57.225929 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:57.225890 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Sep 16 13:51:57.226331 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:57.225949 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" podUID="2698b1b4-3481-41fa-8afc-79551dee1bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Sep 16 13:51:57.720787 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:57.720749 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:57.720953 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:57.720830 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:57.733624 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:57.733584 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:57.762799 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:57.762744 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" podStartSLOduration=2.762724933 podStartE2EDuration="2.762724933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:51:56.468802641 +0000 UTC m=+1264.569615361" watchObservedRunningTime="2026-09-16 13:51:57.762724933 +0000 UTC m=+1265.863537648" Sep 16 13:51:58.225776 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:58.225746 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Sep 16 13:51:58.225946 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:58.225804 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" podUID="2698b1b4-3481-41fa-8afc-79551dee1bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Sep 16 13:51:58.492310 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:58.492228 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:51:59.225649 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:59.225585 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Sep 16 13:51:59.225823 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:51:59.225669 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" podUID="2698b1b4-3481-41fa-8afc-79551dee1bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Sep 16 13:52:00.230299 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:00.230266 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:52:00.230734 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:00.230535 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:52:00.231396 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:00.231378 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 13:52:00.804776 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:00.804738 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:52:02.185228 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.185184 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx"] Sep 16 13:52:02.187894 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.187877 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:02.190827 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.190805 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-973610\"/\"auth-test-model-973610-kserve-self-signed-certs\"" Sep 16 13:52:02.190972 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.190856 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvzqr\" (UniqueName: \"kubernetes.io/projected/39786424-c91c-427a-a41a-1e4758affc48-kube-api-access-hvzqr\") pod \"auth-test-model-973610-kserve-6f57d4989f-sxskx\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:02.191043 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.191011 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-tmp-dir\") pod \"auth-test-model-973610-kserve-6f57d4989f-sxskx\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:02.191104 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.191048 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-model-cache\") pod \"auth-test-model-973610-kserve-6f57d4989f-sxskx\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:02.191104 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.191086 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-home\") pod \"auth-test-model-973610-kserve-6f57d4989f-sxskx\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:02.191219 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.191115 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-dshm\") pod \"auth-test-model-973610-kserve-6f57d4989f-sxskx\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:02.191219 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.191164 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39786424-c91c-427a-a41a-1e4758affc48-tls-certs\") pod \"auth-test-model-973610-kserve-6f57d4989f-sxskx\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:02.192021 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.192000 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-973610\"/\"kube-root-ca.crt\"" Sep 16 13:52:02.192120 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.192005 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-973610\"/\"openshift-service-ca.crt\"" Sep 16 13:52:02.192338 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.192323 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-973610\"/\"default-dockercfg-jksjm\"" Sep 16 13:52:02.200175 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.200146 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx"] Sep 16 13:52:02.221655 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.221618 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:02.292427 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.292385 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvzqr\" (UniqueName: \"kubernetes.io/projected/39786424-c91c-427a-a41a-1e4758affc48-kube-api-access-hvzqr\") pod \"auth-test-model-973610-kserve-6f57d4989f-sxskx\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:02.292637 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.292473 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-tmp-dir\") pod \"auth-test-model-973610-kserve-6f57d4989f-sxskx\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:02.292637 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.292502 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-model-cache\") pod \"auth-test-model-973610-kserve-6f57d4989f-sxskx\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:02.292637 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.292529 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-home\") pod \"auth-test-model-973610-kserve-6f57d4989f-sxskx\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:02.292637 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.292548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-dshm\") pod \"auth-test-model-973610-kserve-6f57d4989f-sxskx\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:02.292637 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.292587 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39786424-c91c-427a-a41a-1e4758affc48-tls-certs\") pod \"auth-test-model-973610-kserve-6f57d4989f-sxskx\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:02.292963 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.292939 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-home\") pod \"auth-test-model-973610-kserve-6f57d4989f-sxskx\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:02.292963 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.292957 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-tmp-dir\") pod \"auth-test-model-973610-kserve-6f57d4989f-sxskx\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:02.293038 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.292978 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-model-cache\") pod \"auth-test-model-973610-kserve-6f57d4989f-sxskx\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:02.295124 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.295106 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-dshm\") pod \"auth-test-model-973610-kserve-6f57d4989f-sxskx\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:02.295316 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.295299 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39786424-c91c-427a-a41a-1e4758affc48-tls-certs\") pod \"auth-test-model-973610-kserve-6f57d4989f-sxskx\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:02.301583 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.301556 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvzqr\" (UniqueName: \"kubernetes.io/projected/39786424-c91c-427a-a41a-1e4758affc48-kube-api-access-hvzqr\") pod \"auth-test-model-973610-kserve-6f57d4989f-sxskx\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:02.499873 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.499833 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:02.773553 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:02.773527 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx"] Sep 16 13:52:02.775416 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:52:02.775387 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39786424_c91c_427a_a41a_1e4758affc48.slice/crio-71d5f2845c8f35c10dba520bb0be4393dc1b96a9eeaa24ae155243181cbdacb5 WatchSource:0}: Error finding container 71d5f2845c8f35c10dba520bb0be4393dc1b96a9eeaa24ae155243181cbdacb5: Status 404 returned error can't find the container with id 71d5f2845c8f35c10dba520bb0be4393dc1b96a9eeaa24ae155243181cbdacb5 Sep 16 13:52:03.487430 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:03.487390 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" event={"ID":"39786424-c91c-427a-a41a-1e4758affc48","Type":"ContainerStarted","Data":"cf1104d9d676d623876d18c2b7281f2c550e83178c328099218c1063540f004e"} Sep 16 13:52:03.487966 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:03.487444 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" event={"ID":"39786424-c91c-427a-a41a-1e4758affc48","Type":"ContainerStarted","Data":"71d5f2845c8f35c10dba520bb0be4393dc1b96a9eeaa24ae155243181cbdacb5"} Sep 16 13:52:03.509843 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:03.509799 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" podStartSLOduration=1.50978288 podStartE2EDuration="1.50978288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:52:03.506873937 +0000 UTC m=+1271.607686654" watchObservedRunningTime="2026-09-16 13:52:03.50978288 +0000 UTC m=+1271.610595594" Sep 16 13:52:05.698508 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:05.698465 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:52:05.698508 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:05.698517 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:52:05.712795 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:05.712762 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:52:06.513015 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:06.512986 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:52:12.500553 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:12.500501 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:12.500553 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:12.500569 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:12.513111 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:12.513083 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:12.543995 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:12.543961 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:52:32.135335 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:32.135249 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:32.940641 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:32.940587 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:33.600370 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:33.600335 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:34.321484 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:34.321438 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:35.055833 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:35.055805 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:35.771256 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:35.771226 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:36.545721 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:36.545687 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:37.312455 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:37.312399 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:38.111169 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:38.111133 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:44.104202 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:44.104157 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:52:47.412154 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:47.412118 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:47.444772 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:47.444739 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l"] Sep 16 13:52:47.444991 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:47.444949 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l" podUID="955b4937-14b8-4bea-95c6-66192463e455" containerName="payload-processing" containerID="cri-o://8fb8b2f6e60a47fad14c04ee3047183adc05e9fa13c8342ebfa2a5c046dcffe8" gracePeriod=30 Sep 16 13:52:47.513767 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:47.513732 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:47.611520 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:47.611483 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:47.697786 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:47.697756 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l" Sep 16 13:52:47.711079 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:47.711048 2580 generic.go:356] "Generic (PLEG): container finished" podID="955b4937-14b8-4bea-95c6-66192463e455" containerID="8fb8b2f6e60a47fad14c04ee3047183adc05e9fa13c8342ebfa2a5c046dcffe8" exitCode=0 Sep 16 13:52:47.711233 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:47.711088 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l" Sep 16 13:52:47.711233 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:47.711088 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l" event={"ID":"955b4937-14b8-4bea-95c6-66192463e455","Type":"ContainerDied","Data":"8fb8b2f6e60a47fad14c04ee3047183adc05e9fa13c8342ebfa2a5c046dcffe8"} Sep 16 13:52:47.711233 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:47.711155 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l" event={"ID":"955b4937-14b8-4bea-95c6-66192463e455","Type":"ContainerDied","Data":"f5c3e687d660d60c715f07df4c000aef281d77e8d46beb814f21652b8571743d"} Sep 16 13:52:47.711233 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:47.711184 2580 scope.go:117] "RemoveContainer" containerID="8fb8b2f6e60a47fad14c04ee3047183adc05e9fa13c8342ebfa2a5c046dcffe8" Sep 16 13:52:47.711418 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:47.711307 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:47.722161 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:47.722094 2580 scope.go:117] "RemoveContainer" containerID="8fb8b2f6e60a47fad14c04ee3047183adc05e9fa13c8342ebfa2a5c046dcffe8" Sep 16 13:52:47.722471 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:52:47.722452 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb8b2f6e60a47fad14c04ee3047183adc05e9fa13c8342ebfa2a5c046dcffe8\": container with ID starting with 8fb8b2f6e60a47fad14c04ee3047183adc05e9fa13c8342ebfa2a5c046dcffe8 not found: ID does not exist" containerID="8fb8b2f6e60a47fad14c04ee3047183adc05e9fa13c8342ebfa2a5c046dcffe8" Sep 16 13:52:47.722567 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:47.722484 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb8b2f6e60a47fad14c04ee3047183adc05e9fa13c8342ebfa2a5c046dcffe8"} err="failed to get container status \"8fb8b2f6e60a47fad14c04ee3047183adc05e9fa13c8342ebfa2a5c046dcffe8\": rpc error: code = NotFound desc = could not find container \"8fb8b2f6e60a47fad14c04ee3047183adc05e9fa13c8342ebfa2a5c046dcffe8\": container with ID starting with 8fb8b2f6e60a47fad14c04ee3047183adc05e9fa13c8342ebfa2a5c046dcffe8 not found: ID does not exist" Sep 16 13:52:47.761212 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:47.761179 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rblv5\" (UniqueName: \"kubernetes.io/projected/955b4937-14b8-4bea-95c6-66192463e455-kube-api-access-rblv5\") pod \"955b4937-14b8-4bea-95c6-66192463e455\" (UID: \"955b4937-14b8-4bea-95c6-66192463e455\") " Sep 16 13:52:47.761408 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:47.761256 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/955b4937-14b8-4bea-95c6-66192463e455-plugins-config-volume\") pod \"955b4937-14b8-4bea-95c6-66192463e455\" (UID: \"955b4937-14b8-4bea-95c6-66192463e455\") " Sep 16 13:52:47.761716 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:47.761680 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/955b4937-14b8-4bea-95c6-66192463e455-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "955b4937-14b8-4bea-95c6-66192463e455" (UID: "955b4937-14b8-4bea-95c6-66192463e455"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:52:47.764038 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:47.764001 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955b4937-14b8-4bea-95c6-66192463e455-kube-api-access-rblv5" (OuterVolumeSpecName: "kube-api-access-rblv5") pod "955b4937-14b8-4bea-95c6-66192463e455" (UID: "955b4937-14b8-4bea-95c6-66192463e455"). InnerVolumeSpecName "kube-api-access-rblv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:52:47.863007 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:47.862952 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rblv5\" (UniqueName: \"kubernetes.io/projected/955b4937-14b8-4bea-95c6-66192463e455-kube-api-access-rblv5\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:52:47.863007 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:47.863000 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/955b4937-14b8-4bea-95c6-66192463e455-plugins-config-volume\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:52:48.054314 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:48.054281 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l"] Sep 16 13:52:48.063665 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:48.063596 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c5dc2482-565d998589-bzl9l"] Sep 16 13:52:48.145466 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:48.145426 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:48.355134 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:48.355052 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="955b4937-14b8-4bea-95c6-66192463e455" path="/var/lib/kubelet/pods/955b4937-14b8-4bea-95c6-66192463e455/volumes" Sep 16 13:52:49.016334 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:49.016304 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:49.726541 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:49.726513 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:50.448542 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:50.448500 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:50.612007 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:50.611972 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:51.106886 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:51.106851 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:51.321441 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:51.321409 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:52.040170 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:52.040131 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:52.774592 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:52.774551 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:56.286898 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:56.286856 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2"] Sep 16 13:52:56.339780 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:52:56.339747 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:53:11.306759 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.306698 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" podUID="13622db8-6ad3-4b22-8bde-beaf54327f81" containerName="main" containerID="cri-o://c637b124f7470d4dd428dfca41c4a7ea5a93dceba0f0b69b6ec68267b6086186" gracePeriod=45 Sep 16 13:53:11.563516 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.563448 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:53:11.596725 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.596695 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-home\") pod \"13622db8-6ad3-4b22-8bde-beaf54327f81\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " Sep 16 13:53:11.596901 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.596732 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdv5n\" (UniqueName: \"kubernetes.io/projected/13622db8-6ad3-4b22-8bde-beaf54327f81-kube-api-access-kdv5n\") pod \"13622db8-6ad3-4b22-8bde-beaf54327f81\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " Sep 16 13:53:11.596901 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.596798 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-model-cache\") pod \"13622db8-6ad3-4b22-8bde-beaf54327f81\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " Sep 16 13:53:11.596901 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.596864 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-tmp-dir\") pod \"13622db8-6ad3-4b22-8bde-beaf54327f81\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " Sep 16 13:53:11.597085 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.596967 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13622db8-6ad3-4b22-8bde-beaf54327f81-tls-certs\") pod \"13622db8-6ad3-4b22-8bde-beaf54327f81\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " Sep 16 13:53:11.597085 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.597011 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-dshm\") pod \"13622db8-6ad3-4b22-8bde-beaf54327f81\" (UID: \"13622db8-6ad3-4b22-8bde-beaf54327f81\") " Sep 16 13:53:11.597202 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.596968 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-home" (OuterVolumeSpecName: "home") pod "13622db8-6ad3-4b22-8bde-beaf54327f81" (UID: "13622db8-6ad3-4b22-8bde-beaf54327f81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:53:11.597202 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.597137 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-model-cache" (OuterVolumeSpecName: "model-cache") pod "13622db8-6ad3-4b22-8bde-beaf54327f81" (UID: "13622db8-6ad3-4b22-8bde-beaf54327f81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:53:11.597345 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.597314 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13622db8-6ad3-4b22-8bde-beaf54327f81" (UID: "13622db8-6ad3-4b22-8bde-beaf54327f81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:53:11.597412 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.597388 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:53:11.597412 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.597402 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:53:11.597528 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.597414 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:53:11.599714 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.599674 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-dshm" (OuterVolumeSpecName: "dshm") pod "13622db8-6ad3-4b22-8bde-beaf54327f81" (UID: "13622db8-6ad3-4b22-8bde-beaf54327f81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:53:11.599939 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.599918 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13622db8-6ad3-4b22-8bde-beaf54327f81-kube-api-access-kdv5n" (OuterVolumeSpecName: "kube-api-access-kdv5n") pod "13622db8-6ad3-4b22-8bde-beaf54327f81" (UID: "13622db8-6ad3-4b22-8bde-beaf54327f81"). InnerVolumeSpecName "kube-api-access-kdv5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:53:11.599939 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.599922 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13622db8-6ad3-4b22-8bde-beaf54327f81-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13622db8-6ad3-4b22-8bde-beaf54327f81" (UID: "13622db8-6ad3-4b22-8bde-beaf54327f81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:53:11.698898 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.698861 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13622db8-6ad3-4b22-8bde-beaf54327f81-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:53:11.698898 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.698890 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdv5n\" (UniqueName: \"kubernetes.io/projected/13622db8-6ad3-4b22-8bde-beaf54327f81-kube-api-access-kdv5n\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:53:11.698898 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.698899 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13622db8-6ad3-4b22-8bde-beaf54327f81-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:53:11.845698 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.845586 2580 generic.go:356] "Generic (PLEG): container finished" podID="13622db8-6ad3-4b22-8bde-beaf54327f81" containerID="c637b124f7470d4dd428dfca41c4a7ea5a93dceba0f0b69b6ec68267b6086186" exitCode=0 Sep 16 13:53:11.845698 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.845644 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" Sep 16 13:53:11.845698 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.845681 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" event={"ID":"13622db8-6ad3-4b22-8bde-beaf54327f81","Type":"ContainerDied","Data":"c637b124f7470d4dd428dfca41c4a7ea5a93dceba0f0b69b6ec68267b6086186"} Sep 16 13:53:11.845917 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.845733 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2" event={"ID":"13622db8-6ad3-4b22-8bde-beaf54327f81","Type":"ContainerDied","Data":"c2e577cfd2a341e551824a1adc214580ce115db2e4d5784261d3a5e40d702948"} Sep 16 13:53:11.845917 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.845753 2580 scope.go:117] "RemoveContainer" containerID="c637b124f7470d4dd428dfca41c4a7ea5a93dceba0f0b69b6ec68267b6086186" Sep 16 13:53:11.854980 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.854964 2580 scope.go:117] "RemoveContainer" containerID="c637b124f7470d4dd428dfca41c4a7ea5a93dceba0f0b69b6ec68267b6086186" Sep 16 13:53:11.855265 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:53:11.855245 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c637b124f7470d4dd428dfca41c4a7ea5a93dceba0f0b69b6ec68267b6086186\": container with ID starting with c637b124f7470d4dd428dfca41c4a7ea5a93dceba0f0b69b6ec68267b6086186 not found: ID does not exist" containerID="c637b124f7470d4dd428dfca41c4a7ea5a93dceba0f0b69b6ec68267b6086186" Sep 16 13:53:11.855330 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.855274 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c637b124f7470d4dd428dfca41c4a7ea5a93dceba0f0b69b6ec68267b6086186"} err="failed to get container status \"c637b124f7470d4dd428dfca41c4a7ea5a93dceba0f0b69b6ec68267b6086186\": rpc error: code = NotFound desc = could not find container \"c637b124f7470d4dd428dfca41c4a7ea5a93dceba0f0b69b6ec68267b6086186\": container with ID starting with c637b124f7470d4dd428dfca41c4a7ea5a93dceba0f0b69b6ec68267b6086186 not found: ID does not exist" Sep 16 13:53:11.868049 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.868023 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2"] Sep 16 13:53:11.871616 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:11.871585 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-c5dc2482/e2e-lifecycle-model-c5dc2482-kserve-7768b4d847-kbtx2"] Sep 16 13:53:12.356167 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:12.356137 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13622db8-6ad3-4b22-8bde-beaf54327f81" path="/var/lib/kubelet/pods/13622db8-6ad3-4b22-8bde-beaf54327f81/volumes" Sep 16 13:53:14.606698 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:14.606599 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:53:26.989049 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:26.989003 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8"] Sep 16 13:53:26.989564 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:26.989447 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13622db8-6ad3-4b22-8bde-beaf54327f81" containerName="main" Sep 16 13:53:26.989564 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:26.989462 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="13622db8-6ad3-4b22-8bde-beaf54327f81" containerName="main" Sep 16 13:53:26.989564 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:26.989474 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="955b4937-14b8-4bea-95c6-66192463e455" containerName="payload-processing" Sep 16 13:53:26.989564 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:26.989481 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="955b4937-14b8-4bea-95c6-66192463e455" containerName="payload-processing" Sep 16 13:53:26.989564 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:26.989548 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="955b4937-14b8-4bea-95c6-66192463e455" containerName="payload-processing" Sep 16 13:53:26.989564 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:26.989556 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="13622db8-6ad3-4b22-8bde-beaf54327f81" containerName="main" Sep 16 13:53:26.992417 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:26.992398 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:26.995189 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:26.995167 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4bb29c\"/\"distinct-2-w4-4bb29c-kserve-self-signed-certs\"" Sep 16 13:53:27.002459 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.002435 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8"] Sep 16 13:53:27.026144 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.026112 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:53:27.062403 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.062367 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-dshm\") pod \"distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:27.062635 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.062418 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a288cad1-3cb3-4953-ba84-471a45108ded-tls-certs\") pod \"distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:27.062635 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.062440 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-tmp-dir\") pod \"distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:27.062635 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.062592 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kskt9\" (UniqueName: \"kubernetes.io/projected/a288cad1-3cb3-4953-ba84-471a45108ded-kube-api-access-kskt9\") pod \"distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:27.062799 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.062654 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-home\") pod \"distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:27.062799 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.062688 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-model-cache\") pod \"distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:27.163772 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.163737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kskt9\" (UniqueName: \"kubernetes.io/projected/a288cad1-3cb3-4953-ba84-471a45108ded-kube-api-access-kskt9\") pod \"distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:27.163986 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.163792 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-home\") pod \"distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:27.163986 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.163828 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-model-cache\") pod \"distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:27.163986 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.163879 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-dshm\") pod \"distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:27.163986 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.163919 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a288cad1-3cb3-4953-ba84-471a45108ded-tls-certs\") pod \"distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:27.163986 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.163945 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-tmp-dir\") pod \"distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:27.164409 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.164385 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-home\") pod \"distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:27.164587 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.164561 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-model-cache\") pod \"distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:27.164808 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.164413 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-tmp-dir\") pod \"distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:27.166697 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.166676 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-dshm\") pod \"distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:27.166833 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.166813 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a288cad1-3cb3-4953-ba84-471a45108ded-tls-certs\") pod \"distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:27.177789 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.177764 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kskt9\" (UniqueName: \"kubernetes.io/projected/a288cad1-3cb3-4953-ba84-471a45108ded-kube-api-access-kskt9\") pod \"distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:27.304330 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.304236 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:27.582485 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.582459 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8"] Sep 16 13:53:27.584636 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:53:27.584576 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda288cad1_3cb3_4953_ba84_471a45108ded.slice/crio-93c91ca8db9b90db94edd015661407d39764fc656aa8d38a4b0fef450520d114 WatchSource:0}: Error finding container 93c91ca8db9b90db94edd015661407d39764fc656aa8d38a4b0fef450520d114: Status 404 returned error can't find the container with id 93c91ca8db9b90db94edd015661407d39764fc656aa8d38a4b0fef450520d114 Sep 16 13:53:27.927694 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.927564 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" event={"ID":"a288cad1-3cb3-4953-ba84-471a45108ded","Type":"ContainerStarted","Data":"d00a7a979cc5a21ff57214258cd699fcf5efcc6c191f2f9ac4688903abf5e5e6"} Sep 16 13:53:27.927694 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.927632 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" event={"ID":"a288cad1-3cb3-4953-ba84-471a45108ded","Type":"ContainerStarted","Data":"93c91ca8db9b90db94edd015661407d39764fc656aa8d38a4b0fef450520d114"} Sep 16 13:53:27.948431 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:27.948371 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" podStartSLOduration=1.94835184 podStartE2EDuration="1.94835184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:53:27.946313736 +0000 UTC m=+1356.047126464" watchObservedRunningTime="2026-09-16 13:53:27.94835184 +0000 UTC m=+1356.049164554" Sep 16 13:53:33.216046 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:33.216013 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:53:33.365351 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:33.365316 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:53:37.107702 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:37.107654 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:53:37.304359 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:37.304323 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:37.304530 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:37.304370 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:37.317116 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:37.317090 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:37.989955 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:37.989925 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:53:39.220882 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:39.220849 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:53:41.475599 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.475563 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77"] Sep 16 13:53:41.479358 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.479333 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:41.483661 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.483639 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8c36a6\"/\"default-dockercfg-9m5ns\"" Sep 16 13:53:41.483823 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.483678 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8c36a6\"/\"openshift-service-ca.crt\"" Sep 16 13:53:41.483823 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.483712 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8c36a6\"/\"auth-test-model-8c36a6-kserve-self-signed-certs\"" Sep 16 13:53:41.483823 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.483639 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8c36a6\"/\"kube-root-ca.crt\"" Sep 16 13:53:41.489237 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.489215 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77"] Sep 16 13:53:41.585793 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.585761 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:53:41.613223 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.613189 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j5m9\" (UniqueName: \"kubernetes.io/projected/12148179-76e7-4a24-bc06-932f8b764b43-kube-api-access-9j5m9\") pod \"auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:41.613392 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.613236 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-home\") pod \"auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:41.613392 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.613262 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12148179-76e7-4a24-bc06-932f8b764b43-tls-certs\") pod \"auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:41.613392 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.613377 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-tmp-dir\") pod \"auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:41.613569 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.613440 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-dshm\") pod \"auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:41.613569 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.613470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-model-cache\") pod \"auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:41.715019 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.714976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-dshm\") pod \"auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:41.715019 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.715019 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-model-cache\") pod \"auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:41.715296 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.715116 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j5m9\" (UniqueName: \"kubernetes.io/projected/12148179-76e7-4a24-bc06-932f8b764b43-kube-api-access-9j5m9\") pod \"auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:41.715296 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.715165 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-home\") pod \"auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:41.715296 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.715199 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12148179-76e7-4a24-bc06-932f8b764b43-tls-certs\") pod \"auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:41.715296 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.715268 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-tmp-dir\") pod \"auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:41.715572 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.715554 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-model-cache\") pod \"auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:41.715669 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.715576 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-home\") pod \"auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:41.715669 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.715597 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-tmp-dir\") pod \"auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:41.717431 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.717408 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-dshm\") pod \"auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:41.717891 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.717873 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12148179-76e7-4a24-bc06-932f8b764b43-tls-certs\") pod \"auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:41.724287 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.724261 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j5m9\" (UniqueName: \"kubernetes.io/projected/12148179-76e7-4a24-bc06-932f8b764b43-kube-api-access-9j5m9\") pod \"auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:41.790743 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:41.790644 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:42.066460 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:42.066436 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77"] Sep 16 13:53:42.068594 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:53:42.068559 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12148179_76e7_4a24_bc06_932f8b764b43.slice/crio-0f408f1949ff37324f5a390aa9cdedbf860092f1a470d2ead8bb76f2d33e9793 WatchSource:0}: Error finding container 0f408f1949ff37324f5a390aa9cdedbf860092f1a470d2ead8bb76f2d33e9793: Status 404 returned error can't find the container with id 0f408f1949ff37324f5a390aa9cdedbf860092f1a470d2ead8bb76f2d33e9793 Sep 16 13:53:43.006582 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:43.006547 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" event={"ID":"12148179-76e7-4a24-bc06-932f8b764b43","Type":"ContainerStarted","Data":"5b89625eb7a716f4633e7fe2b520bc4f353c061246b8bf75750374098b88430c"} Sep 16 13:53:43.007034 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:43.006593 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" event={"ID":"12148179-76e7-4a24-bc06-932f8b764b43","Type":"ContainerStarted","Data":"0f408f1949ff37324f5a390aa9cdedbf860092f1a470d2ead8bb76f2d33e9793"} Sep 16 13:53:43.029068 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:43.025878 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" podStartSLOduration=2.025851228 podStartE2EDuration="2.025851228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:53:43.023425616 +0000 UTC m=+1371.124238332" watchObservedRunningTime="2026-09-16 13:53:43.025851228 +0000 UTC m=+1371.126663944" Sep 16 13:53:51.790984 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:51.790942 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:51.791471 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:51.790999 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:51.804104 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:51.804076 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:53:52.064144 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:53:52.064059 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:54:01.702051 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:01.702012 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:54:13.119515 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:13.119481 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:21.557007 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:21.556965 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs"] Sep 16 13:54:21.557573 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:21.557256 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" podUID="e87c6eae-3609-431f-b1cc-db13e06d7b25" containerName="istio-proxy" containerID="cri-o://8af235dcf9ec1fdf94cd0b61af7e3ccb76004abcd9d0922d6f6283010634a8c4" gracePeriod=30 Sep 16 13:54:21.583668 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:21.583636 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:21.886571 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:21.886476 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:22.849343 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:22.849300 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:22.957288 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:22.957254 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:23.090479 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:23.090440 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:23.559503 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:23.559470 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:54:23.743113 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:23.743078 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:23.829141 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:23.829059 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:23.928791 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:23.928761 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:25.248233 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:25.248202 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:26.813921 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.813896 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:54:26.881741 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.881663 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-data\") pod \"e87c6eae-3609-431f-b1cc-db13e06d7b25\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " Sep 16 13:54:26.881741 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.881708 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-podinfo\") pod \"e87c6eae-3609-431f-b1cc-db13e06d7b25\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " Sep 16 13:54:26.881741 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.881737 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e87c6eae-3609-431f-b1cc-db13e06d7b25-istiod-ca-cert\") pod \"e87c6eae-3609-431f-b1cc-db13e06d7b25\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " Sep 16 13:54:26.881999 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.881815 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r97d9\" (UniqueName: \"kubernetes.io/projected/e87c6eae-3609-431f-b1cc-db13e06d7b25-kube-api-access-r97d9\") pod \"e87c6eae-3609-431f-b1cc-db13e06d7b25\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " Sep 16 13:54:26.881999 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.881848 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-workload-certs\") pod \"e87c6eae-3609-431f-b1cc-db13e06d7b25\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " Sep 16 13:54:26.881999 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.881881 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-token\") pod \"e87c6eae-3609-431f-b1cc-db13e06d7b25\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " Sep 16 13:54:26.881999 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.881906 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-workload-socket\") pod \"e87c6eae-3609-431f-b1cc-db13e06d7b25\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " Sep 16 13:54:26.881999 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.881936 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-envoy\") pod \"e87c6eae-3609-431f-b1cc-db13e06d7b25\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " Sep 16 13:54:26.881999 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.881962 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-credential-socket\") pod \"e87c6eae-3609-431f-b1cc-db13e06d7b25\" (UID: \"e87c6eae-3609-431f-b1cc-db13e06d7b25\") " Sep 16 13:54:26.881999 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.881968 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-data" (OuterVolumeSpecName: "istio-data") pod "e87c6eae-3609-431f-b1cc-db13e06d7b25" (UID: "e87c6eae-3609-431f-b1cc-db13e06d7b25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:54:26.882332 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.882265 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e87c6eae-3609-431f-b1cc-db13e06d7b25-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e87c6eae-3609-431f-b1cc-db13e06d7b25" (UID: "e87c6eae-3609-431f-b1cc-db13e06d7b25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:54:26.882397 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.882363 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e87c6eae-3609-431f-b1cc-db13e06d7b25" (UID: "e87c6eae-3609-431f-b1cc-db13e06d7b25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:54:26.882449 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.882400 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e87c6eae-3609-431f-b1cc-db13e06d7b25" (UID: "e87c6eae-3609-431f-b1cc-db13e06d7b25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:54:26.882556 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.882530 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e87c6eae-3609-431f-b1cc-db13e06d7b25-istiod-ca-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:54:26.882817 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.882562 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-workload-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:54:26.882817 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.882576 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-workload-socket\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:54:26.882817 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.882587 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-data\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:54:26.882817 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.882586 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e87c6eae-3609-431f-b1cc-db13e06d7b25" (UID: "e87c6eae-3609-431f-b1cc-db13e06d7b25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:54:26.884231 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.884206 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e87c6eae-3609-431f-b1cc-db13e06d7b25" (UID: "e87c6eae-3609-431f-b1cc-db13e06d7b25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 13:54:26.884348 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.884314 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e87c6eae-3609-431f-b1cc-db13e06d7b25" (UID: "e87c6eae-3609-431f-b1cc-db13e06d7b25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:54:26.884396 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.884345 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-token" (OuterVolumeSpecName: "istio-token") pod "e87c6eae-3609-431f-b1cc-db13e06d7b25" (UID: "e87c6eae-3609-431f-b1cc-db13e06d7b25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:54:26.884476 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.884460 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87c6eae-3609-431f-b1cc-db13e06d7b25-kube-api-access-r97d9" (OuterVolumeSpecName: "kube-api-access-r97d9") pod "e87c6eae-3609-431f-b1cc-db13e06d7b25" (UID: "e87c6eae-3609-431f-b1cc-db13e06d7b25"). InnerVolumeSpecName "kube-api-access-r97d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:54:26.983896 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.983859 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r97d9\" (UniqueName: \"kubernetes.io/projected/e87c6eae-3609-431f-b1cc-db13e06d7b25-kube-api-access-r97d9\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:54:26.983896 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.983890 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-token\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:54:26.983896 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.983900 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-envoy\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:54:26.984195 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.983908 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e87c6eae-3609-431f-b1cc-db13e06d7b25-credential-socket\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:54:26.984195 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:26.983918 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e87c6eae-3609-431f-b1cc-db13e06d7b25-istio-podinfo\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:54:27.237748 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:27.237712 2580 generic.go:356] "Generic (PLEG): container finished" podID="e87c6eae-3609-431f-b1cc-db13e06d7b25" containerID="8af235dcf9ec1fdf94cd0b61af7e3ccb76004abcd9d0922d6f6283010634a8c4" exitCode=0 Sep 16 13:54:27.237938 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:27.237759 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" Sep 16 13:54:27.237938 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:27.237800 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" event={"ID":"e87c6eae-3609-431f-b1cc-db13e06d7b25","Type":"ContainerDied","Data":"8af235dcf9ec1fdf94cd0b61af7e3ccb76004abcd9d0922d6f6283010634a8c4"} Sep 16 13:54:27.237938 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:27.237877 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs" event={"ID":"e87c6eae-3609-431f-b1cc-db13e06d7b25","Type":"ContainerDied","Data":"81ad428d108354cdf3e9988e748346ec6409fdf8bbe6a2f0a68c1317e5c59d05"} Sep 16 13:54:27.237938 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:27.237900 2580 scope.go:117] "RemoveContainer" containerID="8af235dcf9ec1fdf94cd0b61af7e3ccb76004abcd9d0922d6f6283010634a8c4" Sep 16 13:54:27.248067 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:27.248048 2580 scope.go:117] "RemoveContainer" containerID="74af86032ffb684be380adddb7e77d3cd0fd251b26fd6e265fc3ba2fbc5209aa" Sep 16 13:54:27.256402 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:27.256381 2580 scope.go:117] "RemoveContainer" containerID="8af235dcf9ec1fdf94cd0b61af7e3ccb76004abcd9d0922d6f6283010634a8c4" Sep 16 13:54:27.256683 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:54:27.256663 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8af235dcf9ec1fdf94cd0b61af7e3ccb76004abcd9d0922d6f6283010634a8c4\": container with ID starting with 8af235dcf9ec1fdf94cd0b61af7e3ccb76004abcd9d0922d6f6283010634a8c4 not found: ID does not exist" containerID="8af235dcf9ec1fdf94cd0b61af7e3ccb76004abcd9d0922d6f6283010634a8c4" Sep 16 13:54:27.256763 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:27.256696 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8af235dcf9ec1fdf94cd0b61af7e3ccb76004abcd9d0922d6f6283010634a8c4"} err="failed to get container status \"8af235dcf9ec1fdf94cd0b61af7e3ccb76004abcd9d0922d6f6283010634a8c4\": rpc error: code = NotFound desc = could not find container \"8af235dcf9ec1fdf94cd0b61af7e3ccb76004abcd9d0922d6f6283010634a8c4\": container with ID starting with 8af235dcf9ec1fdf94cd0b61af7e3ccb76004abcd9d0922d6f6283010634a8c4 not found: ID does not exist" Sep 16 13:54:27.256763 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:27.256720 2580 scope.go:117] "RemoveContainer" containerID="74af86032ffb684be380adddb7e77d3cd0fd251b26fd6e265fc3ba2fbc5209aa" Sep 16 13:54:27.257081 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:54:27.257022 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74af86032ffb684be380adddb7e77d3cd0fd251b26fd6e265fc3ba2fbc5209aa\": container with ID starting with 74af86032ffb684be380adddb7e77d3cd0fd251b26fd6e265fc3ba2fbc5209aa not found: ID does not exist" containerID="74af86032ffb684be380adddb7e77d3cd0fd251b26fd6e265fc3ba2fbc5209aa" Sep 16 13:54:27.257185 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:27.257089 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74af86032ffb684be380adddb7e77d3cd0fd251b26fd6e265fc3ba2fbc5209aa"} err="failed to get container status \"74af86032ffb684be380adddb7e77d3cd0fd251b26fd6e265fc3ba2fbc5209aa\": rpc error: code = NotFound desc = could not find container \"74af86032ffb684be380adddb7e77d3cd0fd251b26fd6e265fc3ba2fbc5209aa\": container with ID starting with 74af86032ffb684be380adddb7e77d3cd0fd251b26fd6e265fc3ba2fbc5209aa not found: ID does not exist" Sep 16 13:54:27.260216 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:27.260194 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs"] Sep 16 13:54:27.265902 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:27.265872 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c5dc2482-openshift-default-5dfc494ddd-bfdzs"] Sep 16 13:54:28.355848 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:28.355813 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e87c6eae-3609-431f-b1cc-db13e06d7b25" path="/var/lib/kubelet/pods/e87c6eae-3609-431f-b1cc-db13e06d7b25/volumes" Sep 16 13:54:31.124863 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:31.124829 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:31.210936 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:31.210902 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:31.861374 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:31.861339 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:37.055625 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:37.055560 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:37.161395 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:37.161357 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:37.517286 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:37.517253 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:38.020071 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:38.020040 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:38.122830 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:38.122794 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:38.218431 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:38.218397 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:48.743184 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:48.743101 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:48.985646 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:48.985581 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:49.905805 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:54:49.905769 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:55:02.902220 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:02.902186 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:09.920631 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:09.920573 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:12.742986 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:12.742949 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:15.890104 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:15.890070 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:15.992400 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:15.992361 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:16.202021 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.201927 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:16.505368 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.505333 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29"] Sep 16 13:55:16.505959 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.505938 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e87c6eae-3609-431f-b1cc-db13e06d7b25" containerName="istio-proxy" Sep 16 13:55:16.506048 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.505965 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e87c6eae-3609-431f-b1cc-db13e06d7b25" containerName="istio-proxy" Sep 16 13:55:16.506108 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.506094 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e87c6eae-3609-431f-b1cc-db13e06d7b25" containerName="istio-proxy" Sep 16 13:55:16.506146 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.506116 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e87c6eae-3609-431f-b1cc-db13e06d7b25" containerName="istio-proxy" Sep 16 13:55:16.510787 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.510762 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.514472 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.514437 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-bdd04e-openshift-default-dockercfg-4tkk6\"" Sep 16 13:55:16.523822 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.523798 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29"] Sep 16 13:55:16.539545 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.539516 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:16.596001 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.595970 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-podinfo\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.596001 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.596004 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-token\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.596270 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.596122 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38a7e4d3-28e7-4cca-b729-de6502258dd8-istiod-ca-cert\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.596270 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.596147 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs26h\" (UniqueName: \"kubernetes.io/projected/38a7e4d3-28e7-4cca-b729-de6502258dd8-kube-api-access-cs26h\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.596270 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.596198 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-envoy\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.596270 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.596231 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-workload-socket\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.596441 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.596300 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-credential-socket\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.596441 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.596361 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-data\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.596441 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.596406 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-workload-certs\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.697715 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.697666 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-podinfo\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.697715 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.697724 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-token\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.698000 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.697813 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38a7e4d3-28e7-4cca-b729-de6502258dd8-istiod-ca-cert\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.698000 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.697845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs26h\" (UniqueName: \"kubernetes.io/projected/38a7e4d3-28e7-4cca-b729-de6502258dd8-kube-api-access-cs26h\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.698000 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.697883 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-envoy\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.698000 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.697923 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-workload-socket\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.698000 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.697953 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-credential-socket\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.698264 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.698004 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-data\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.698264 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.698043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-workload-certs\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.698552 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.698481 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-workload-socket\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.698697 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.698648 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38a7e4d3-28e7-4cca-b729-de6502258dd8-istiod-ca-cert\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.698697 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.698652 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-workload-certs\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.698791 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.698500 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-credential-socket\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.698791 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.698772 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-data\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.700297 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.700274 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-envoy\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.700403 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.700366 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-podinfo\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.706854 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.706830 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs26h\" (UniqueName: \"kubernetes.io/projected/38a7e4d3-28e7-4cca-b729-de6502258dd8-kube-api-access-cs26h\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.707349 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.707331 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-token\") pod \"e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.823116 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.823014 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:16.834092 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:16.834063 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:17.110656 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:17.110448 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29"] Sep 16 13:55:17.113427 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:55:17.113392 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a7e4d3_28e7_4cca_b729_de6502258dd8.slice/crio-be81d34990afa979567fa292a38b3b1a4b395d8726aab792238962b6c86d6bb1 WatchSource:0}: Error finding container be81d34990afa979567fa292a38b3b1a4b395d8726aab792238962b6c86d6bb1: Status 404 returned error can't find the container with id be81d34990afa979567fa292a38b3b1a4b395d8726aab792238962b6c86d6bb1 Sep 16 13:55:17.115929 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:17.115886 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:55:17.116007 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:17.115985 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:55:17.116053 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:17.116034 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:55:17.495282 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:17.495246 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" event={"ID":"38a7e4d3-28e7-4cca-b729-de6502258dd8","Type":"ContainerStarted","Data":"55f55c90d4209a2eb3d1baee5bdbfd973b3eaf931757be7d334ba249136ba028"} Sep 16 13:55:17.495563 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:17.495317 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" event={"ID":"38a7e4d3-28e7-4cca-b729-de6502258dd8","Type":"ContainerStarted","Data":"be81d34990afa979567fa292a38b3b1a4b395d8726aab792238962b6c86d6bb1"} Sep 16 13:55:17.519268 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:17.519218 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" podStartSLOduration=1.519203789 podStartE2EDuration="1.519203789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:17.516576211 +0000 UTC m=+1465.617388925" watchObservedRunningTime="2026-09-16 13:55:17.519203789 +0000 UTC m=+1465.620016498" Sep 16 13:55:17.823482 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:17.823386 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:18.828576 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:18.828542 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:18.839421 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:18.839391 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:19.493565 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:19.493528 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:19.506133 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:19.506104 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:19.507102 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:19.507076 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:55:19.585335 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:19.585306 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:19.844761 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:19.844661 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:20.644153 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:20.644115 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:21.536629 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:21.536576 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:22.231630 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:22.231576 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:22.942265 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:22.942221 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:23.697186 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:23.697152 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:24.530706 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:24.530672 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:24.629248 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:24.629215 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:24.857072 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:24.856978 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:24.975925 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:24.975890 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:25.079359 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:25.079319 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:26.668749 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:26.668710 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:55:45.345868 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:45.345836 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:45.491334 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:45.491298 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:46.834943 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:46.834904 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:47.533356 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:47.533323 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:48.397674 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:48.397646 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:48.930501 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:48.930459 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:49.619898 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:49.619862 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:49.800359 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:49.800326 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:50.102406 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.102367 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:50.418321 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.418234 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:50.467378 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.467344 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l"] Sep 16 13:55:50.467763 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.467748 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e87c6eae-3609-431f-b1cc-db13e06d7b25" containerName="istio-proxy" Sep 16 13:55:50.467763 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.467763 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e87c6eae-3609-431f-b1cc-db13e06d7b25" containerName="istio-proxy" Sep 16 13:55:50.471031 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.471014 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.473505 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.473483 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-59a5a8-openshift-default-dockercfg-gbxll\"" Sep 16 13:55:50.482048 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.482020 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l"] Sep 16 13:55:50.600394 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.600355 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:50.651213 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.651180 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-data\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.651213 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.651216 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-token\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.651710 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.651240 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-envoy\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.651710 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.651266 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istiod-ca-cert\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.651710 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.651350 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-workload-socket\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.651710 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.651436 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-workload-certs\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.651710 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.651455 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jfjk\" (UniqueName: \"kubernetes.io/projected/ed40eb9d-9f9a-496c-b298-ae975d3090a5-kube-api-access-8jfjk\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.651710 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.651471 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-credential-socket\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.651710 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.651515 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-podinfo\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.698137 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.698025 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:50.753148 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.753106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-podinfo\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.753357 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.753161 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-data\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.753357 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.753184 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-token\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.753357 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.753215 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-envoy\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.753357 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.753251 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istiod-ca-cert\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.753357 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.753313 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-workload-socket\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.753661 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.753405 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-workload-certs\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.753661 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.753430 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jfjk\" (UniqueName: \"kubernetes.io/projected/ed40eb9d-9f9a-496c-b298-ae975d3090a5-kube-api-access-8jfjk\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.753661 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.753456 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-credential-socket\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.753833 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.753763 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-workload-socket\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.753891 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.753860 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-workload-certs\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.753891 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.753878 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-data\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.753989 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.753889 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-credential-socket\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.754060 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.754041 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istiod-ca-cert\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.755701 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.755683 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-envoy\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.755957 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.755943 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-podinfo\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.764746 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.764721 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-token\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.764929 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.764912 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jfjk\" (UniqueName: \"kubernetes.io/projected/ed40eb9d-9f9a-496c-b298-ae975d3090a5-kube-api-access-8jfjk\") pod \"e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:50.782896 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:50.782871 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:51.066855 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:51.066824 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l"] Sep 16 13:55:51.071117 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:51.071071 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:55:51.071224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:51.071158 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:55:51.071224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:51.071196 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:55:51.338397 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:51.338291 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:51.679087 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:51.678985 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" event={"ID":"ed40eb9d-9f9a-496c-b298-ae975d3090a5","Type":"ContainerStarted","Data":"50a9f50a570c6fa290b153cd0c34522ae6e887423c49dea5f813539db515b5ea"} Sep 16 13:55:51.679087 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:51.679045 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" event={"ID":"ed40eb9d-9f9a-496c-b298-ae975d3090a5","Type":"ContainerStarted","Data":"f50cbf1ed563539233dd763efd66298a39d32cb7d667d4db4c6c2e36e0a67d76"} Sep 16 13:55:51.707295 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:51.707236 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" podStartSLOduration=1.707218991 podStartE2EDuration="1.707218991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:51.704694416 +0000 UTC m=+1499.805507131" watchObservedRunningTime="2026-09-16 13:55:51.707218991 +0000 UTC m=+1499.808031705" Sep 16 13:55:51.783286 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:51.783237 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:52.092812 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:52.092781 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:52.627891 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:52.627858 2580 scope.go:117] "RemoveContainer" containerID="a7b7812857f37731a2408061abeb37f1518f9116aef1fb39ba13194e725dec63" Sep 16 13:55:52.807538 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:52.807487 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:53.120925 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:53.120886 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:53.697268 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:53.697237 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:53.698263 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:53.698242 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:55:53.708699 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:53.708674 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:53.793244 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:53.793212 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:54.053243 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:54.053203 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:55.044484 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:55.044455 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:55.742948 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:55.742917 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:56.783149 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:56.783112 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:57.270090 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:57.270056 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:58.034701 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:58.034661 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:58.536833 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:58.536791 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46ccbd-b9chr"] Sep 16 13:55:58.541801 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:58.541775 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46ccbd-b9chr" Sep 16 13:55:58.545707 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:58.545682 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:55:58.546038 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:58.545695 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-thk5m\"" Sep 16 13:55:58.549111 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:58.549084 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46ccbd-b9chr"] Sep 16 13:55:58.636074 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:58.636034 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a39d759-dd16-4a98-b856-bde94b387b59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-46ccbd-b9chr\" (UID: \"3a39d759-dd16-4a98-b856-bde94b387b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46ccbd-b9chr" Sep 16 13:55:58.736963 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:58.736918 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a39d759-dd16-4a98-b856-bde94b387b59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-46ccbd-b9chr\" (UID: \"3a39d759-dd16-4a98-b856-bde94b387b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46ccbd-b9chr" Sep 16 13:55:58.737540 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:58.737513 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a39d759-dd16-4a98-b856-bde94b387b59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-46ccbd-b9chr\" (UID: \"3a39d759-dd16-4a98-b856-bde94b387b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46ccbd-b9chr" Sep 16 13:55:58.856390 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:58.856284 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46ccbd-b9chr" Sep 16 13:55:59.133109 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:59.133082 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46ccbd-b9chr"] Sep 16 13:55:59.135429 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:55:59.135400 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a39d759_dd16_4a98_b856_bde94b387b59.slice/crio-764f29791f645cfe6fd33edbf6de5b62d3072a59ac4d8b2a4de28575a2883b06 WatchSource:0}: Error finding container 764f29791f645cfe6fd33edbf6de5b62d3072a59ac4d8b2a4de28575a2883b06: Status 404 returned error can't find the container with id 764f29791f645cfe6fd33edbf6de5b62d3072a59ac4d8b2a4de28575a2883b06 Sep 16 13:55:59.732161 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:59.732129 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46ccbd-b9chr" event={"ID":"3a39d759-dd16-4a98-b856-bde94b387b59","Type":"ContainerStarted","Data":"17d7f8914706329984193e85adef87e6ba93008bf8cc8bd9a3429daa99890812"} Sep 16 13:55:59.732379 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:59.732181 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46ccbd-b9chr" event={"ID":"3a39d759-dd16-4a98-b856-bde94b387b59","Type":"ContainerStarted","Data":"764f29791f645cfe6fd33edbf6de5b62d3072a59ac4d8b2a4de28575a2883b06"} Sep 16 13:55:59.748311 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:55:59.748230 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46ccbd-b9chr" podStartSLOduration=1.748212528 podStartE2EDuration="1.748212528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:59.747471503 +0000 UTC m=+1507.848284222" watchObservedRunningTime="2026-09-16 13:55:59.748212528 +0000 UTC m=+1507.849025243" Sep 16 13:56:00.692034 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:00.691943 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:00.739355 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:00.739321 2580 generic.go:356] "Generic (PLEG): container finished" podID="38a7e4d3-28e7-4cca-b729-de6502258dd8" containerID="55f55c90d4209a2eb3d1baee5bdbfd973b3eaf931757be7d334ba249136ba028" exitCode=0 Sep 16 13:56:00.739548 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:00.739394 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" event={"ID":"38a7e4d3-28e7-4cca-b729-de6502258dd8","Type":"ContainerDied","Data":"55f55c90d4209a2eb3d1baee5bdbfd973b3eaf931757be7d334ba249136ba028"} Sep 16 13:56:00.739928 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:00.739911 2580 scope.go:117] "RemoveContainer" containerID="55f55c90d4209a2eb3d1baee5bdbfd973b3eaf931757be7d334ba249136ba028" Sep 16 13:56:00.741167 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:00.741143 2580 generic.go:356] "Generic (PLEG): container finished" podID="3a39d759-dd16-4a98-b856-bde94b387b59" containerID="17d7f8914706329984193e85adef87e6ba93008bf8cc8bd9a3429daa99890812" exitCode=0 Sep 16 13:56:00.741263 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:00.741191 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46ccbd-b9chr" event={"ID":"3a39d759-dd16-4a98-b856-bde94b387b59","Type":"ContainerDied","Data":"17d7f8914706329984193e85adef87e6ba93008bf8cc8bd9a3429daa99890812"} Sep 16 13:56:00.741263 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:00.741226 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:56:00.741346 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:00.741287 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:56:00.741346 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:00.741318 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:56:01.545090 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:01.545054 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:01.748712 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:01.748674 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" event={"ID":"38a7e4d3-28e7-4cca-b729-de6502258dd8","Type":"ContainerStarted","Data":"7b29b75e802ed23c5a59e4e976009f5fa7e020b2bebc43acb935a7fc97da5a4c"} Sep 16 13:56:01.823436 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:01.823402 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:56:01.824476 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:01.824447 2580 patch_prober.go:28] interesting pod/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Sep 16 13:56:01.824638 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:01.824508 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" podUID="38a7e4d3-28e7-4cca-b729-de6502258dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Sep 16 13:56:01.893274 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:01.893088 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46ccbd-b9chr" Sep 16 13:56:01.975919 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:01.975871 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a39d759-dd16-4a98-b856-bde94b387b59-maas-api-service-ca\") pod \"3a39d759-dd16-4a98-b856-bde94b387b59\" (UID: \"3a39d759-dd16-4a98-b856-bde94b387b59\") " Sep 16 13:56:01.980636 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:01.976763 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a39d759-dd16-4a98-b856-bde94b387b59-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a39d759-dd16-4a98-b856-bde94b387b59" (UID: "3a39d759-dd16-4a98-b856-bde94b387b59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:56:02.078458 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:02.078351 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a39d759-dd16-4a98-b856-bde94b387b59-maas-api-service-ca\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:02.316730 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:02.316689 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:02.755668 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:02.755637 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46ccbd-b9chr" Sep 16 13:56:02.755668 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:02.755644 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46ccbd-b9chr" event={"ID":"3a39d759-dd16-4a98-b856-bde94b387b59","Type":"ContainerDied","Data":"764f29791f645cfe6fd33edbf6de5b62d3072a59ac4d8b2a4de28575a2883b06"} Sep 16 13:56:02.756111 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:02.755692 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="764f29791f645cfe6fd33edbf6de5b62d3072a59ac4d8b2a4de28575a2883b06" Sep 16 13:56:02.824287 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:02.824251 2580 patch_prober.go:28] interesting pod/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Sep 16 13:56:02.824469 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:02.824306 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" podUID="38a7e4d3-28e7-4cca-b729-de6502258dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Sep 16 13:56:03.210643 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:03.210525 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:03.508949 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:03.508917 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9"] Sep 16 13:56:03.542716 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:03.542683 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g"] Sep 16 13:56:03.554424 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:03.554393 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr"] Sep 16 13:56:03.632567 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:03.632532 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:03.824366 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:03.824273 2580 patch_prober.go:28] interesting pod/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Sep 16 13:56:03.824366 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:03.824341 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" podUID="38a7e4d3-28e7-4cca-b729-de6502258dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Sep 16 13:56:04.150206 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:04.150121 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:04.506720 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:04.506685 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:56:04.823988 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:04.823892 2580 patch_prober.go:28] interesting pod/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Sep 16 13:56:04.823988 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:04.823955 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" podUID="38a7e4d3-28e7-4cca-b729-de6502258dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Sep 16 13:56:04.948486 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:04.948448 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:05.505556 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:05.505520 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:56:05.650467 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:05.650414 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:05.823834 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:05.823734 2580 patch_prober.go:28] interesting pod/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Sep 16 13:56:05.823834 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:05.823801 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" podUID="38a7e4d3-28e7-4cca-b729-de6502258dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Sep 16 13:56:06.388020 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:06.387976 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:06.823395 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:06.823356 2580 patch_prober.go:28] interesting pod/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Sep 16 13:56:06.823585 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:06.823417 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" podUID="38a7e4d3-28e7-4cca-b729-de6502258dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Sep 16 13:56:07.828234 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:07.828200 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:56:07.829338 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:07.829307 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:56:08.861020 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:08.860989 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:13.202453 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:13.202407 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:14.478707 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.478669 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:14.479159 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.478849 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46ccbd-29826105-5q2dl"] Sep 16 13:56:14.480904 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.480883 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-46ccbd-29826105-5q2dl"] Sep 16 13:56:14.515280 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.515250 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc"] Sep 16 13:56:14.515484 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.515454 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc" podUID="2741e4f1-17b2-4882-b6f2-1bcbf877666c" containerName="payload-pre-processing" containerID="cri-o://75ebf02d014bb61af2d0c0a03d35e57e79fa1158d2c14e45499bd0ccac69027b" gracePeriod=30 Sep 16 13:56:14.576306 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.576275 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:14.678668 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.678235 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:14.688985 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.688950 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46ccbd-b9chr"] Sep 16 13:56:14.691117 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.691089 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-46ccbd-b9chr"] Sep 16 13:56:14.765313 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.765289 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc" Sep 16 13:56:14.807253 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.807226 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:14.818386 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.818355 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4h7zg\" (UniqueName: \"kubernetes.io/projected/2741e4f1-17b2-4882-b6f2-1bcbf877666c-kube-api-access-4h7zg\") pod \"2741e4f1-17b2-4882-b6f2-1bcbf877666c\" (UID: \"2741e4f1-17b2-4882-b6f2-1bcbf877666c\") " Sep 16 13:56:14.818538 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.818409 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2741e4f1-17b2-4882-b6f2-1bcbf877666c-plugins-config-volume\") pod \"2741e4f1-17b2-4882-b6f2-1bcbf877666c\" (UID: \"2741e4f1-17b2-4882-b6f2-1bcbf877666c\") " Sep 16 13:56:14.818804 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.818777 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2741e4f1-17b2-4882-b6f2-1bcbf877666c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2741e4f1-17b2-4882-b6f2-1bcbf877666c" (UID: "2741e4f1-17b2-4882-b6f2-1bcbf877666c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:56:14.820659 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.820637 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2741e4f1-17b2-4882-b6f2-1bcbf877666c-kube-api-access-4h7zg" (OuterVolumeSpecName: "kube-api-access-4h7zg") pod "2741e4f1-17b2-4882-b6f2-1bcbf877666c" (UID: "2741e4f1-17b2-4882-b6f2-1bcbf877666c"). InnerVolumeSpecName "kube-api-access-4h7zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:56:14.832915 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.832889 2580 generic.go:356] "Generic (PLEG): container finished" podID="2741e4f1-17b2-4882-b6f2-1bcbf877666c" containerID="75ebf02d014bb61af2d0c0a03d35e57e79fa1158d2c14e45499bd0ccac69027b" exitCode=0 Sep 16 13:56:14.833054 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.832947 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc" Sep 16 13:56:14.833054 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.832982 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc" event={"ID":"2741e4f1-17b2-4882-b6f2-1bcbf877666c","Type":"ContainerDied","Data":"75ebf02d014bb61af2d0c0a03d35e57e79fa1158d2c14e45499bd0ccac69027b"} Sep 16 13:56:14.833162 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.833062 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc" event={"ID":"2741e4f1-17b2-4882-b6f2-1bcbf877666c","Type":"ContainerDied","Data":"2ca9463eddf728edff863064b5c9e366486ee1bd7c57d58819a2224e781d1423"} Sep 16 13:56:14.833162 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.833094 2580 scope.go:117] "RemoveContainer" containerID="75ebf02d014bb61af2d0c0a03d35e57e79fa1158d2c14e45499bd0ccac69027b" Sep 16 13:56:14.852143 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.852111 2580 scope.go:117] "RemoveContainer" containerID="75ebf02d014bb61af2d0c0a03d35e57e79fa1158d2c14e45499bd0ccac69027b" Sep 16 13:56:14.853003 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:56:14.852976 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ebf02d014bb61af2d0c0a03d35e57e79fa1158d2c14e45499bd0ccac69027b\": container with ID starting with 75ebf02d014bb61af2d0c0a03d35e57e79fa1158d2c14e45499bd0ccac69027b not found: ID does not exist" containerID="75ebf02d014bb61af2d0c0a03d35e57e79fa1158d2c14e45499bd0ccac69027b" Sep 16 13:56:14.853123 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.853013 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ebf02d014bb61af2d0c0a03d35e57e79fa1158d2c14e45499bd0ccac69027b"} err="failed to get container status \"75ebf02d014bb61af2d0c0a03d35e57e79fa1158d2c14e45499bd0ccac69027b\": rpc error: code = NotFound desc = could not find container \"75ebf02d014bb61af2d0c0a03d35e57e79fa1158d2c14e45499bd0ccac69027b\": container with ID starting with 75ebf02d014bb61af2d0c0a03d35e57e79fa1158d2c14e45499bd0ccac69027b not found: ID does not exist" Sep 16 13:56:14.870466 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.870441 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc"] Sep 16 13:56:14.887133 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.887109 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-46ccbd-6f48d85c5d-djsgc"] Sep 16 13:56:14.920253 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.920222 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h7zg\" (UniqueName: \"kubernetes.io/projected/2741e4f1-17b2-4882-b6f2-1bcbf877666c-kube-api-access-4h7zg\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:14.920253 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:14.920249 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2741e4f1-17b2-4882-b6f2-1bcbf877666c-plugins-config-volume\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:15.123472 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:15.123367 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:15.218400 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:15.218362 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:15.928341 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:15.928314 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:16.355537 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:16.355503 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2741e4f1-17b2-4882-b6f2-1bcbf877666c" path="/var/lib/kubelet/pods/2741e4f1-17b2-4882-b6f2-1bcbf877666c/volumes" Sep 16 13:56:16.355916 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:16.355901 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a39d759-dd16-4a98-b856-bde94b387b59" path="/var/lib/kubelet/pods/3a39d759-dd16-4a98-b856-bde94b387b59/volumes" Sep 16 13:56:16.356158 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:16.356145 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92a54a7c-7643-4ce6-a2a5-e68ee98b66f0" path="/var/lib/kubelet/pods/92a54a7c-7643-4ce6-a2a5-e68ee98b66f0/volumes" Sep 16 13:56:16.669508 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:16.669414 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:17.494592 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:17.494558 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:18.332757 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.332725 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:18.529523 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.529477 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" podUID="fbd672f1-f0ba-4787-8666-67f462e15c84" containerName="main" containerID="cri-o://ec613a09259ec71c541f3e233d0f26004f5b64efc247ade5c66468fc73fd3429" gracePeriod=45 Sep 16 13:56:18.589521 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.589410 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" podUID="4f129b28-ba14-4640-b632-b9eb00e9485e" containerName="main" containerID="cri-o://1b231c1ffaaf37b02b6df54bbdc686e94c3871513796aca21511bc8418258cfc" gracePeriod=45 Sep 16 13:56:18.603633 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.603587 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" podUID="4471abf1-d68c-4ac9-adc0-b375ac39c347" containerName="main" containerID="cri-o://49dc3ecf4c34e9c87c2e00f8553358b483803f412a0fea01a716f61c4eefa91d" gracePeriod=45 Sep 16 13:56:18.861812 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.860491 2580 generic.go:356] "Generic (PLEG): container finished" podID="4471abf1-d68c-4ac9-adc0-b375ac39c347" containerID="49dc3ecf4c34e9c87c2e00f8553358b483803f412a0fea01a716f61c4eefa91d" exitCode=0 Sep 16 13:56:18.861812 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.860575 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" event={"ID":"4471abf1-d68c-4ac9-adc0-b375ac39c347","Type":"ContainerDied","Data":"49dc3ecf4c34e9c87c2e00f8553358b483803f412a0fea01a716f61c4eefa91d"} Sep 16 13:56:18.862651 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.862446 2580 generic.go:356] "Generic (PLEG): container finished" podID="4f129b28-ba14-4640-b632-b9eb00e9485e" containerID="1b231c1ffaaf37b02b6df54bbdc686e94c3871513796aca21511bc8418258cfc" exitCode=0 Sep 16 13:56:18.862846 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.862657 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" event={"ID":"4f129b28-ba14-4640-b632-b9eb00e9485e","Type":"ContainerDied","Data":"1b231c1ffaaf37b02b6df54bbdc686e94c3871513796aca21511bc8418258cfc"} Sep 16 13:56:18.862846 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.862687 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" event={"ID":"4f129b28-ba14-4640-b632-b9eb00e9485e","Type":"ContainerDied","Data":"41e2b3a17c49f568c0c95da384983c22fd041051af6c3e12d543f7b15268f737"} Sep 16 13:56:18.862846 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.862702 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41e2b3a17c49f568c0c95da384983c22fd041051af6c3e12d543f7b15268f737" Sep 16 13:56:18.865068 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.865046 2580 generic.go:356] "Generic (PLEG): container finished" podID="fbd672f1-f0ba-4787-8666-67f462e15c84" containerID="ec613a09259ec71c541f3e233d0f26004f5b64efc247ade5c66468fc73fd3429" exitCode=0 Sep 16 13:56:18.865157 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.865115 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" event={"ID":"fbd672f1-f0ba-4787-8666-67f462e15c84","Type":"ContainerDied","Data":"ec613a09259ec71c541f3e233d0f26004f5b64efc247ade5c66468fc73fd3429"} Sep 16 13:56:18.871238 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.871215 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:56:18.895504 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.895475 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:56:18.900133 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.900111 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:56:18.967900 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.967859 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-model-cache\") pod \"4f129b28-ba14-4640-b632-b9eb00e9485e\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " Sep 16 13:56:18.968133 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.967909 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-home\") pod \"fbd672f1-f0ba-4787-8666-67f462e15c84\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " Sep 16 13:56:18.968133 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.967926 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-tmp-dir\") pod \"fbd672f1-f0ba-4787-8666-67f462e15c84\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " Sep 16 13:56:18.968133 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.967952 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-dshm\") pod \"fbd672f1-f0ba-4787-8666-67f462e15c84\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " Sep 16 13:56:18.968133 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.967994 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f129b28-ba14-4640-b632-b9eb00e9485e-tls-certs\") pod \"4f129b28-ba14-4640-b632-b9eb00e9485e\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " Sep 16 13:56:18.968133 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.968024 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-home\") pod \"4f129b28-ba14-4640-b632-b9eb00e9485e\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " Sep 16 13:56:18.968133 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.968060 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cst99\" (UniqueName: \"kubernetes.io/projected/fbd672f1-f0ba-4787-8666-67f462e15c84-kube-api-access-cst99\") pod \"fbd672f1-f0ba-4787-8666-67f462e15c84\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " Sep 16 13:56:18.968133 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.968099 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jv5k\" (UniqueName: \"kubernetes.io/projected/4471abf1-d68c-4ac9-adc0-b375ac39c347-kube-api-access-8jv5k\") pod \"4471abf1-d68c-4ac9-adc0-b375ac39c347\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " Sep 16 13:56:18.968133 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.968129 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd672f1-f0ba-4787-8666-67f462e15c84-tls-certs\") pod \"fbd672f1-f0ba-4787-8666-67f462e15c84\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " Sep 16 13:56:18.968548 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.968165 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-dshm\") pod \"4471abf1-d68c-4ac9-adc0-b375ac39c347\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " Sep 16 13:56:18.968548 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.968182 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-home" (OuterVolumeSpecName: "home") pod "fbd672f1-f0ba-4787-8666-67f462e15c84" (UID: "fbd672f1-f0ba-4787-8666-67f462e15c84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:18.968548 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.968204 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-tmp-dir\") pod \"4471abf1-d68c-4ac9-adc0-b375ac39c347\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " Sep 16 13:56:18.968548 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.968238 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fbd672f1-f0ba-4787-8666-67f462e15c84" (UID: "fbd672f1-f0ba-4787-8666-67f462e15c84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:18.968548 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.968263 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-model-cache\") pod \"4471abf1-d68c-4ac9-adc0-b375ac39c347\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " Sep 16 13:56:18.968548 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.968294 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-dshm\") pod \"4f129b28-ba14-4640-b632-b9eb00e9485e\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " Sep 16 13:56:18.968548 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.968329 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qphxl\" (UniqueName: \"kubernetes.io/projected/4f129b28-ba14-4640-b632-b9eb00e9485e-kube-api-access-qphxl\") pod \"4f129b28-ba14-4640-b632-b9eb00e9485e\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " Sep 16 13:56:18.968548 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.968428 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4471abf1-d68c-4ac9-adc0-b375ac39c347" (UID: "4471abf1-d68c-4ac9-adc0-b375ac39c347"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:18.968548 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.968443 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-home" (OuterVolumeSpecName: "home") pod "4f129b28-ba14-4640-b632-b9eb00e9485e" (UID: "4f129b28-ba14-4640-b632-b9eb00e9485e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:18.968548 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.968464 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f129b28-ba14-4640-b632-b9eb00e9485e" (UID: "4f129b28-ba14-4640-b632-b9eb00e9485e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:18.969268 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.969113 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-model-cache" (OuterVolumeSpecName: "model-cache") pod "4471abf1-d68c-4ac9-adc0-b375ac39c347" (UID: "4471abf1-d68c-4ac9-adc0-b375ac39c347"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:18.969268 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.969246 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-model-cache\") pod \"fbd672f1-f0ba-4787-8666-67f462e15c84\" (UID: \"fbd672f1-f0ba-4787-8666-67f462e15c84\") " Sep 16 13:56:18.969387 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.969291 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4471abf1-d68c-4ac9-adc0-b375ac39c347-tls-certs\") pod \"4471abf1-d68c-4ac9-adc0-b375ac39c347\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " Sep 16 13:56:18.969387 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.969366 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-tmp-dir\") pod \"4f129b28-ba14-4640-b632-b9eb00e9485e\" (UID: \"4f129b28-ba14-4640-b632-b9eb00e9485e\") " Sep 16 13:56:18.969495 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.969396 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-home\") pod \"4471abf1-d68c-4ac9-adc0-b375ac39c347\" (UID: \"4471abf1-d68c-4ac9-adc0-b375ac39c347\") " Sep 16 13:56:18.970120 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.969939 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-model-cache" (OuterVolumeSpecName: "model-cache") pod "fbd672f1-f0ba-4787-8666-67f462e15c84" (UID: "fbd672f1-f0ba-4787-8666-67f462e15c84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:18.970253 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.970148 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-home" (OuterVolumeSpecName: "home") pod "4471abf1-d68c-4ac9-adc0-b375ac39c347" (UID: "4471abf1-d68c-4ac9-adc0-b375ac39c347"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:18.970326 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.970241 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f129b28-ba14-4640-b632-b9eb00e9485e" (UID: "4f129b28-ba14-4640-b632-b9eb00e9485e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:18.970726 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.970568 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:18.970726 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.970596 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:18.970726 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.970632 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:18.970726 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.970646 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:18.970726 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.970659 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:18.970726 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.970671 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:18.970726 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.970682 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:18.970726 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.970694 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:18.970726 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.970705 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:18.971402 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.971225 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-dshm" (OuterVolumeSpecName: "dshm") pod "fbd672f1-f0ba-4787-8666-67f462e15c84" (UID: "fbd672f1-f0ba-4787-8666-67f462e15c84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:18.971402 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.971321 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd672f1-f0ba-4787-8666-67f462e15c84-kube-api-access-cst99" (OuterVolumeSpecName: "kube-api-access-cst99") pod "fbd672f1-f0ba-4787-8666-67f462e15c84" (UID: "fbd672f1-f0ba-4787-8666-67f462e15c84"). InnerVolumeSpecName "kube-api-access-cst99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:56:18.971654 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.971591 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-dshm" (OuterVolumeSpecName: "dshm") pod "4471abf1-d68c-4ac9-adc0-b375ac39c347" (UID: "4471abf1-d68c-4ac9-adc0-b375ac39c347"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:18.971654 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.971599 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f129b28-ba14-4640-b632-b9eb00e9485e-kube-api-access-qphxl" (OuterVolumeSpecName: "kube-api-access-qphxl") pod "4f129b28-ba14-4640-b632-b9eb00e9485e" (UID: "4f129b28-ba14-4640-b632-b9eb00e9485e"). InnerVolumeSpecName "kube-api-access-qphxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:56:18.972242 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.972220 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4471abf1-d68c-4ac9-adc0-b375ac39c347-kube-api-access-8jv5k" (OuterVolumeSpecName: "kube-api-access-8jv5k") pod "4471abf1-d68c-4ac9-adc0-b375ac39c347" (UID: "4471abf1-d68c-4ac9-adc0-b375ac39c347"). InnerVolumeSpecName "kube-api-access-8jv5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:56:18.972834 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.972815 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-dshm" (OuterVolumeSpecName: "dshm") pod "4f129b28-ba14-4640-b632-b9eb00e9485e" (UID: "4f129b28-ba14-4640-b632-b9eb00e9485e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:18.972921 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.972907 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4471abf1-d68c-4ac9-adc0-b375ac39c347-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4471abf1-d68c-4ac9-adc0-b375ac39c347" (UID: "4471abf1-d68c-4ac9-adc0-b375ac39c347"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:56:18.973165 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.973135 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f129b28-ba14-4640-b632-b9eb00e9485e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f129b28-ba14-4640-b632-b9eb00e9485e" (UID: "4f129b28-ba14-4640-b632-b9eb00e9485e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:56:18.973279 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:18.973263 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbd672f1-f0ba-4787-8666-67f462e15c84-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fbd672f1-f0ba-4787-8666-67f462e15c84" (UID: "fbd672f1-f0ba-4787-8666-67f462e15c84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:56:19.039246 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.039209 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:19.071552 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.071520 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd672f1-f0ba-4787-8666-67f462e15c84-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:19.071552 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.071550 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f129b28-ba14-4640-b632-b9eb00e9485e-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:19.071762 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.071561 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cst99\" (UniqueName: \"kubernetes.io/projected/fbd672f1-f0ba-4787-8666-67f462e15c84-kube-api-access-cst99\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:19.071762 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.071573 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jv5k\" (UniqueName: \"kubernetes.io/projected/4471abf1-d68c-4ac9-adc0-b375ac39c347-kube-api-access-8jv5k\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:19.071762 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.071583 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd672f1-f0ba-4787-8666-67f462e15c84-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:19.071762 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.071591 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4471abf1-d68c-4ac9-adc0-b375ac39c347-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:19.071762 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.071598 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f129b28-ba14-4640-b632-b9eb00e9485e-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:19.071762 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.071621 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qphxl\" (UniqueName: \"kubernetes.io/projected/4f129b28-ba14-4640-b632-b9eb00e9485e-kube-api-access-qphxl\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:19.071762 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.071635 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4471abf1-d68c-4ac9-adc0-b375ac39c347-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:19.746771 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.746738 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:19.871521 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.871430 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" Sep 16 13:56:19.871521 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.871454 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9" event={"ID":"fbd672f1-f0ba-4787-8666-67f462e15c84","Type":"ContainerDied","Data":"32c99320782d70359e1b1f98793f5be67225eb4c08900585ff46af4b62767d2c"} Sep 16 13:56:19.871521 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.871498 2580 scope.go:117] "RemoveContainer" containerID="ec613a09259ec71c541f3e233d0f26004f5b64efc247ade5c66468fc73fd3429" Sep 16 13:56:19.873048 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.873022 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" Sep 16 13:56:19.873190 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.873028 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g" Sep 16 13:56:19.873190 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.873061 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr" event={"ID":"4471abf1-d68c-4ac9-adc0-b375ac39c347","Type":"ContainerDied","Data":"a1bf21ab22351ddd88f38c48e452b15b50e21cf2cfdf9d95998fdb515b2ad6cc"} Sep 16 13:56:19.883103 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.882974 2580 scope.go:117] "RemoveContainer" containerID="49dc3ecf4c34e9c87c2e00f8553358b483803f412a0fea01a716f61c4eefa91d" Sep 16 13:56:19.898628 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.898586 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9"] Sep 16 13:56:19.904254 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.904229 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/distinct-2-w0-46ccbd-kserve-5cb74b6c8-mnvh9"] Sep 16 13:56:19.914386 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.914351 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g"] Sep 16 13:56:19.916277 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.916257 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/premium-w0-46ccbd-kserve-67784f9874-92p9g"] Sep 16 13:56:19.928183 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.928155 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr"] Sep 16 13:56:19.931909 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:19.931885 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/simulator-w0-46ccbd-kserve-8498c66847-pwxtr"] Sep 16 13:56:20.356373 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:20.356337 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4471abf1-d68c-4ac9-adc0-b375ac39c347" path="/var/lib/kubelet/pods/4471abf1-d68c-4ac9-adc0-b375ac39c347/volumes" Sep 16 13:56:20.356773 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:20.356760 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f129b28-ba14-4640-b632-b9eb00e9485e" path="/var/lib/kubelet/pods/4f129b28-ba14-4640-b632-b9eb00e9485e/volumes" Sep 16 13:56:20.357118 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:20.357106 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbd672f1-f0ba-4787-8666-67f462e15c84" path="/var/lib/kubelet/pods/fbd672f1-f0ba-4787-8666-67f462e15c84/volumes" Sep 16 13:56:20.727311 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:20.727284 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:21.577355 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:21.577318 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:21.679197 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:21.679161 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:21.932294 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:21.932211 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:22.135295 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:22.135261 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:22.233426 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:22.233390 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:23.008097 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:23.008066 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:23.617954 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:23.617920 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:23.719518 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:23.719486 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:23.842843 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:23.842804 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:24.519263 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:24.519227 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:25.324820 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:25.324782 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:26.124026 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:26.123968 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:26.881172 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:26.881135 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:26.923758 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:26.923719 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd04e-xllp4"] Sep 16 13:56:26.924116 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:26.924104 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f129b28-ba14-4640-b632-b9eb00e9485e" containerName="main" Sep 16 13:56:26.924162 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:26.924118 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f129b28-ba14-4640-b632-b9eb00e9485e" containerName="main" Sep 16 13:56:26.924162 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:26.924132 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbd672f1-f0ba-4787-8666-67f462e15c84" containerName="main" Sep 16 13:56:26.924162 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:26.924138 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbd672f1-f0ba-4787-8666-67f462e15c84" containerName="main" Sep 16 13:56:26.924162 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:26.924156 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a39d759-dd16-4a98-b856-bde94b387b59" containerName="revoke-keys" Sep 16 13:56:26.924162 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:26.924161 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a39d759-dd16-4a98-b856-bde94b387b59" containerName="revoke-keys" Sep 16 13:56:26.924318 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:26.924170 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2741e4f1-17b2-4882-b6f2-1bcbf877666c" containerName="payload-pre-processing" Sep 16 13:56:26.924318 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:26.924175 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2741e4f1-17b2-4882-b6f2-1bcbf877666c" containerName="payload-pre-processing" Sep 16 13:56:26.924318 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:26.924181 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4471abf1-d68c-4ac9-adc0-b375ac39c347" containerName="main" Sep 16 13:56:26.924318 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:26.924186 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4471abf1-d68c-4ac9-adc0-b375ac39c347" containerName="main" Sep 16 13:56:26.924318 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:26.924242 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbd672f1-f0ba-4787-8666-67f462e15c84" containerName="main" Sep 16 13:56:26.924318 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:26.924249 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2741e4f1-17b2-4882-b6f2-1bcbf877666c" containerName="payload-pre-processing" Sep 16 13:56:26.924318 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:26.924257 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a39d759-dd16-4a98-b856-bde94b387b59" containerName="revoke-keys" Sep 16 13:56:26.924318 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:26.924265 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4471abf1-d68c-4ac9-adc0-b375ac39c347" containerName="main" Sep 16 13:56:26.924318 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:26.924271 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f129b28-ba14-4640-b632-b9eb00e9485e" containerName="main" Sep 16 13:56:26.927352 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:26.927332 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd04e-xllp4" Sep 16 13:56:26.940057 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:26.940037 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:56:26.940976 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:26.940959 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-thk5m\"" Sep 16 13:56:26.944558 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:26.944532 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd04e-xllp4"] Sep 16 13:56:27.054385 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:27.054352 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0307f3a7-ac9c-46ff-84a4-1af797e72e9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdd04e-xllp4\" (UID: \"0307f3a7-ac9c-46ff-84a4-1af797e72e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd04e-xllp4" Sep 16 13:56:27.156055 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:27.155957 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0307f3a7-ac9c-46ff-84a4-1af797e72e9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdd04e-xllp4\" (UID: \"0307f3a7-ac9c-46ff-84a4-1af797e72e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd04e-xllp4" Sep 16 13:56:27.156564 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:27.156543 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0307f3a7-ac9c-46ff-84a4-1af797e72e9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdd04e-xllp4\" (UID: \"0307f3a7-ac9c-46ff-84a4-1af797e72e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd04e-xllp4" Sep 16 13:56:27.237965 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:27.237920 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd04e-xllp4" Sep 16 13:56:27.532404 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:27.532362 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd04e-xllp4"] Sep 16 13:56:27.534858 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:56:27.534831 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0307f3a7_ac9c_46ff_84a4_1af797e72e9b.slice/crio-f529207643560e77bdde0d1d14ae47d3ac12746a0518cab8864be5b4cc108c98 WatchSource:0}: Error finding container f529207643560e77bdde0d1d14ae47d3ac12746a0518cab8864be5b4cc108c98: Status 404 returned error can't find the container with id f529207643560e77bdde0d1d14ae47d3ac12746a0518cab8864be5b4cc108c98 Sep 16 13:56:27.696860 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:27.696825 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:27.926682 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:27.926641 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd04e-xllp4" event={"ID":"0307f3a7-ac9c-46ff-84a4-1af797e72e9b","Type":"ContainerStarted","Data":"0a95338b6266bcc7b6f6ac8817b8e9770fecc9041cd922175bfbb91f56a9a15b"} Sep 16 13:56:27.926882 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:27.926707 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd04e-xllp4" event={"ID":"0307f3a7-ac9c-46ff-84a4-1af797e72e9b","Type":"ContainerStarted","Data":"f529207643560e77bdde0d1d14ae47d3ac12746a0518cab8864be5b4cc108c98"} Sep 16 13:56:27.946734 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:27.946675 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd04e-xllp4" podStartSLOduration=1.94665729 podStartE2EDuration="1.94665729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:56:27.944654569 +0000 UTC m=+1536.045467285" watchObservedRunningTime="2026-09-16 13:56:27.94665729 +0000 UTC m=+1536.047470006" Sep 16 13:56:28.422484 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:28.422451 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:28.933938 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:28.933902 2580 generic.go:356] "Generic (PLEG): container finished" podID="0307f3a7-ac9c-46ff-84a4-1af797e72e9b" containerID="0a95338b6266bcc7b6f6ac8817b8e9770fecc9041cd922175bfbb91f56a9a15b" exitCode=0 Sep 16 13:56:28.934129 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:28.933990 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd04e-xllp4" event={"ID":"0307f3a7-ac9c-46ff-84a4-1af797e72e9b","Type":"ContainerDied","Data":"0a95338b6266bcc7b6f6ac8817b8e9770fecc9041cd922175bfbb91f56a9a15b"} Sep 16 13:56:29.126182 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:29.126150 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:29.912004 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:29.911971 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:30.080203 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:30.080169 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd04e-xllp4" Sep 16 13:56:30.190309 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:30.190214 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0307f3a7-ac9c-46ff-84a4-1af797e72e9b-maas-api-service-ca\") pod \"0307f3a7-ac9c-46ff-84a4-1af797e72e9b\" (UID: \"0307f3a7-ac9c-46ff-84a4-1af797e72e9b\") " Sep 16 13:56:30.190557 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:30.190531 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0307f3a7-ac9c-46ff-84a4-1af797e72e9b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0307f3a7-ac9c-46ff-84a4-1af797e72e9b" (UID: "0307f3a7-ac9c-46ff-84a4-1af797e72e9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:56:30.292179 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:30.292134 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0307f3a7-ac9c-46ff-84a4-1af797e72e9b-maas-api-service-ca\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:30.839450 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:30.839422 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:30.945744 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:30.945712 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd04e-xllp4" Sep 16 13:56:30.945744 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:30.945730 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd04e-xllp4" event={"ID":"0307f3a7-ac9c-46ff-84a4-1af797e72e9b","Type":"ContainerDied","Data":"f529207643560e77bdde0d1d14ae47d3ac12746a0518cab8864be5b4cc108c98"} Sep 16 13:56:30.948125 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:30.945774 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f529207643560e77bdde0d1d14ae47d3ac12746a0518cab8864be5b4cc108c98" Sep 16 13:56:31.609463 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:31.609437 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:31.952805 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:31.952717 2580 generic.go:356] "Generic (PLEG): container finished" podID="ed40eb9d-9f9a-496c-b298-ae975d3090a5" containerID="50a9f50a570c6fa290b153cd0c34522ae6e887423c49dea5f813539db515b5ea" exitCode=0 Sep 16 13:56:31.953337 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:31.952795 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" event={"ID":"ed40eb9d-9f9a-496c-b298-ae975d3090a5","Type":"ContainerDied","Data":"50a9f50a570c6fa290b153cd0c34522ae6e887423c49dea5f813539db515b5ea"} Sep 16 13:56:31.953401 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:31.953385 2580 scope.go:117] "RemoveContainer" containerID="50a9f50a570c6fa290b153cd0c34522ae6e887423c49dea5f813539db515b5ea" Sep 16 13:56:31.954644 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:31.954593 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:56:31.954749 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:31.954683 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:56:31.954749 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:31.954716 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 13:56:32.408155 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:32.408124 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:32.509473 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:32.509439 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:32.960294 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:32.960249 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" event={"ID":"ed40eb9d-9f9a-496c-b298-ae975d3090a5","Type":"ContainerStarted","Data":"f9531c4291ce43f9edbbf7c32ac6e0e1a2f651b776decb7de2881b2f51aa2ec1"} Sep 16 13:56:33.148805 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:33.148754 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:33.783937 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:33.783902 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:56:33.784357 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:33.784256 2580 patch_prober.go:28] interesting pod/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Sep 16 13:56:33.784547 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:33.784522 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" podUID="ed40eb9d-9f9a-496c-b298-ae975d3090a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Sep 16 13:56:34.713939 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:34.713907 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:56:34.784331 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:34.784281 2580 patch_prober.go:28] interesting pod/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Sep 16 13:56:34.784542 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:34.784354 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" podUID="ed40eb9d-9f9a-496c-b298-ae975d3090a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Sep 16 13:56:35.783626 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:35.783560 2580 patch_prober.go:28] interesting pod/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Sep 16 13:56:35.784156 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:35.783654 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" podUID="ed40eb9d-9f9a-496c-b298-ae975d3090a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Sep 16 13:56:36.783624 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:36.783558 2580 patch_prober.go:28] interesting pod/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Sep 16 13:56:36.784158 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:36.783659 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" podUID="ed40eb9d-9f9a-496c-b298-ae975d3090a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Sep 16 13:56:37.069209 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:37.069058 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:37.783595 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:37.783541 2580 patch_prober.go:28] interesting pod/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Sep 16 13:56:37.783847 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:37.783644 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" podUID="ed40eb9d-9f9a-496c-b298-ae975d3090a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Sep 16 13:56:38.697812 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:38.697772 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:56:38.788516 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:38.788480 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:56:39.002211 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:39.002183 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:56:40.116179 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:40.116147 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:40.680196 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:40.680155 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:40.781991 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:40.781955 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:41.220506 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:41.220471 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:42.864486 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:42.864450 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:42.938329 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:42.938154 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:43.003572 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:43.003537 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:43.069476 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:43.069439 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd04e-xllp4"] Sep 16 13:56:43.072811 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:43.072788 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd04e-xllp4"] Sep 16 13:56:43.084568 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:43.084540 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:43.437237 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:43.437205 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:43.546503 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:43.546460 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:44.355406 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:44.355363 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0307f3a7-ac9c-46ff-84a4-1af797e72e9b" path="/var/lib/kubelet/pods/0307f3a7-ac9c-46ff-84a4-1af797e72e9b/volumes" Sep 16 13:56:44.464769 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:44.464738 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:45.211044 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:45.211007 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:46.056300 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:46.056267 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:46.838623 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:46.838574 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:47.303487 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:47.303454 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:47.630207 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:47.630126 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:48.300811 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:48.300767 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:49.609099 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:49.609061 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:49.709746 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:49.709718 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:49.858323 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:49.858292 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29"] Sep 16 13:56:49.858659 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:49.858588 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" podUID="38a7e4d3-28e7-4cca-b729-de6502258dd8" containerName="istio-proxy" containerID="cri-o://7b29b75e802ed23c5a59e4e976009f5fa7e020b2bebc43acb935a7fc97da5a4c" gracePeriod=30 Sep 16 13:56:49.878491 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:49.878409 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:50.196340 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:50.196251 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59a5a8-w9hgh"] Sep 16 13:56:50.196660 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:50.196646 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0307f3a7-ac9c-46ff-84a4-1af797e72e9b" containerName="revoke-keys" Sep 16 13:56:50.196717 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:50.196661 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0307f3a7-ac9c-46ff-84a4-1af797e72e9b" containerName="revoke-keys" Sep 16 13:56:50.196754 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:50.196718 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0307f3a7-ac9c-46ff-84a4-1af797e72e9b" containerName="revoke-keys" Sep 16 13:56:50.199930 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:50.199913 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59a5a8-w9hgh" Sep 16 13:56:50.202832 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:50.202808 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:56:50.202964 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:50.202853 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-thk5m\"" Sep 16 13:56:50.209126 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:50.209098 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59a5a8-w9hgh"] Sep 16 13:56:50.306178 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:50.306135 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/953a2440-d7fc-4825-b31c-b2ed5720a0cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-59a5a8-w9hgh\" (UID: \"953a2440-d7fc-4825-b31c-b2ed5720a0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59a5a8-w9hgh" Sep 16 13:56:50.407753 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:50.407714 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/953a2440-d7fc-4825-b31c-b2ed5720a0cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-59a5a8-w9hgh\" (UID: \"953a2440-d7fc-4825-b31c-b2ed5720a0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59a5a8-w9hgh" Sep 16 13:56:50.408360 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:50.408340 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/953a2440-d7fc-4825-b31c-b2ed5720a0cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-59a5a8-w9hgh\" (UID: \"953a2440-d7fc-4825-b31c-b2ed5720a0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59a5a8-w9hgh" Sep 16 13:56:50.512086 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:50.512053 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59a5a8-w9hgh" Sep 16 13:56:50.526342 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:50.526310 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:50.788457 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:50.788435 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59a5a8-w9hgh"] Sep 16 13:56:50.790722 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:56:50.790683 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod953a2440_d7fc_4825_b31c_b2ed5720a0cd.slice/crio-6ded32cf75d24c4998933b45389eda8535843bbbcf3387658a6dd2e5833fd9f5 WatchSource:0}: Error finding container 6ded32cf75d24c4998933b45389eda8535843bbbcf3387658a6dd2e5833fd9f5: Status 404 returned error can't find the container with id 6ded32cf75d24c4998933b45389eda8535843bbbcf3387658a6dd2e5833fd9f5 Sep 16 13:56:51.068425 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:51.068392 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59a5a8-w9hgh" event={"ID":"953a2440-d7fc-4825-b31c-b2ed5720a0cd","Type":"ContainerStarted","Data":"6ded32cf75d24c4998933b45389eda8535843bbbcf3387658a6dd2e5833fd9f5"} Sep 16 13:56:51.229224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:51.229190 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:52.037206 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:52.037171 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:52.075419 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:52.075384 2580 generic.go:356] "Generic (PLEG): container finished" podID="953a2440-d7fc-4825-b31c-b2ed5720a0cd" containerID="4efdfa847da52983bd7c1944fac6a3b9bcafbc14e44fff20865d83254a4fb490" exitCode=0 Sep 16 13:56:52.075617 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:52.075465 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59a5a8-w9hgh" event={"ID":"953a2440-d7fc-4825-b31c-b2ed5720a0cd","Type":"ContainerDied","Data":"4efdfa847da52983bd7c1944fac6a3b9bcafbc14e44fff20865d83254a4fb490"} Sep 16 13:56:52.648770 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:52.648738 2580 scope.go:117] "RemoveContainer" containerID="c47a6f1595735cdef9f2b125768ae34afb6954b0b26015c3b4907b8c34bd6c80" Sep 16 13:56:52.657143 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:52.657127 2580 scope.go:117] "RemoveContainer" containerID="1b231c1ffaaf37b02b6df54bbdc686e94c3871513796aca21511bc8418258cfc" Sep 16 13:56:52.665050 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:52.665028 2580 scope.go:117] "RemoveContainer" containerID="3521a0c6aa979bdfa5da0f9f349b36d76ef72fe14b1290b25b8ef63fcff00fb9" Sep 16 13:56:52.733900 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:52.733868 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:52.831597 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:52.831558 2580 patch_prober.go:28] interesting pod/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 13:56:52.831800 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:52.831643 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" podUID="38a7e4d3-28e7-4cca-b729-de6502258dd8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 13:56:53.213116 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:53.213092 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59a5a8-w9hgh" Sep 16 13:56:53.338225 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:53.338141 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/953a2440-d7fc-4825-b31c-b2ed5720a0cd-maas-api-service-ca\") pod \"953a2440-d7fc-4825-b31c-b2ed5720a0cd\" (UID: \"953a2440-d7fc-4825-b31c-b2ed5720a0cd\") " Sep 16 13:56:53.338534 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:53.338509 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/953a2440-d7fc-4825-b31c-b2ed5720a0cd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "953a2440-d7fc-4825-b31c-b2ed5720a0cd" (UID: "953a2440-d7fc-4825-b31c-b2ed5720a0cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:56:53.422951 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:53.422924 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:53.439318 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:53.439278 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/953a2440-d7fc-4825-b31c-b2ed5720a0cd-maas-api-service-ca\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:54.087934 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:54.087901 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59a5a8-w9hgh" Sep 16 13:56:54.087934 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:54.087928 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59a5a8-w9hgh" event={"ID":"953a2440-d7fc-4825-b31c-b2ed5720a0cd","Type":"ContainerDied","Data":"6ded32cf75d24c4998933b45389eda8535843bbbcf3387658a6dd2e5833fd9f5"} Sep 16 13:56:54.088281 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:54.087995 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ded32cf75d24c4998933b45389eda8535843bbbcf3387658a6dd2e5833fd9f5" Sep 16 13:56:54.308353 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:54.308323 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:55.095455 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.095418 2580 generic.go:356] "Generic (PLEG): container finished" podID="38a7e4d3-28e7-4cca-b729-de6502258dd8" containerID="7b29b75e802ed23c5a59e4e976009f5fa7e020b2bebc43acb935a7fc97da5a4c" exitCode=0 Sep 16 13:56:55.095455 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.095442 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" event={"ID":"38a7e4d3-28e7-4cca-b729-de6502258dd8","Type":"ContainerDied","Data":"7b29b75e802ed23c5a59e4e976009f5fa7e020b2bebc43acb935a7fc97da5a4c"} Sep 16 13:56:55.095672 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.095503 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" event={"ID":"38a7e4d3-28e7-4cca-b729-de6502258dd8","Type":"ContainerDied","Data":"be81d34990afa979567fa292a38b3b1a4b395d8726aab792238962b6c86d6bb1"} Sep 16 13:56:55.095672 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.095523 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be81d34990afa979567fa292a38b3b1a4b395d8726aab792238962b6c86d6bb1" Sep 16 13:56:55.095672 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.095537 2580 scope.go:117] "RemoveContainer" containerID="55f55c90d4209a2eb3d1baee5bdbfd973b3eaf931757be7d334ba249136ba028" Sep 16 13:56:55.110911 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:56:55.110887 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55f55c90d4209a2eb3d1baee5bdbfd973b3eaf931757be7d334ba249136ba028\": container with ID starting with 55f55c90d4209a2eb3d1baee5bdbfd973b3eaf931757be7d334ba249136ba028 not found: ID does not exist" containerID="55f55c90d4209a2eb3d1baee5bdbfd973b3eaf931757be7d334ba249136ba028" Sep 16 13:56:55.114349 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.114331 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:56:55.157043 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.156949 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:55.255813 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.255780 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-data\") pod \"38a7e4d3-28e7-4cca-b729-de6502258dd8\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " Sep 16 13:56:55.256016 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.255827 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-podinfo\") pod \"38a7e4d3-28e7-4cca-b729-de6502258dd8\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " Sep 16 13:56:55.256016 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.255856 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38a7e4d3-28e7-4cca-b729-de6502258dd8-istiod-ca-cert\") pod \"38a7e4d3-28e7-4cca-b729-de6502258dd8\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " Sep 16 13:56:55.256016 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.255871 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-envoy\") pod \"38a7e4d3-28e7-4cca-b729-de6502258dd8\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " Sep 16 13:56:55.256016 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.255926 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-credential-socket\") pod \"38a7e4d3-28e7-4cca-b729-de6502258dd8\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " Sep 16 13:56:55.256016 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.255951 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-workload-certs\") pod \"38a7e4d3-28e7-4cca-b729-de6502258dd8\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " Sep 16 13:56:55.256016 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.255976 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-token\") pod \"38a7e4d3-28e7-4cca-b729-de6502258dd8\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " Sep 16 13:56:55.256349 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.256044 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cs26h\" (UniqueName: \"kubernetes.io/projected/38a7e4d3-28e7-4cca-b729-de6502258dd8-kube-api-access-cs26h\") pod \"38a7e4d3-28e7-4cca-b729-de6502258dd8\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " Sep 16 13:56:55.256349 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.256078 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-workload-socket\") pod \"38a7e4d3-28e7-4cca-b729-de6502258dd8\" (UID: \"38a7e4d3-28e7-4cca-b729-de6502258dd8\") " Sep 16 13:56:55.256349 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.256132 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-data" (OuterVolumeSpecName: "istio-data") pod "38a7e4d3-28e7-4cca-b729-de6502258dd8" (UID: "38a7e4d3-28e7-4cca-b729-de6502258dd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:55.256349 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.256220 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a7e4d3-28e7-4cca-b729-de6502258dd8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "38a7e4d3-28e7-4cca-b729-de6502258dd8" (UID: "38a7e4d3-28e7-4cca-b729-de6502258dd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:56:55.256349 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.256225 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "38a7e4d3-28e7-4cca-b729-de6502258dd8" (UID: "38a7e4d3-28e7-4cca-b729-de6502258dd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:55.256813 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.256418 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "38a7e4d3-28e7-4cca-b729-de6502258dd8" (UID: "38a7e4d3-28e7-4cca-b729-de6502258dd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:55.256813 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.256495 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-data\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:55.256813 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.256515 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38a7e4d3-28e7-4cca-b729-de6502258dd8-istiod-ca-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:55.256813 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.256527 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-workload-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:55.256813 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.256496 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "38a7e4d3-28e7-4cca-b729-de6502258dd8" (UID: "38a7e4d3-28e7-4cca-b729-de6502258dd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:55.258328 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.258300 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "38a7e4d3-28e7-4cca-b729-de6502258dd8" (UID: "38a7e4d3-28e7-4cca-b729-de6502258dd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:55.258328 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.258306 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a7e4d3-28e7-4cca-b729-de6502258dd8-kube-api-access-cs26h" (OuterVolumeSpecName: "kube-api-access-cs26h") pod "38a7e4d3-28e7-4cca-b729-de6502258dd8" (UID: "38a7e4d3-28e7-4cca-b729-de6502258dd8"). InnerVolumeSpecName "kube-api-access-cs26h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:56:55.258531 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.258374 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "38a7e4d3-28e7-4cca-b729-de6502258dd8" (UID: "38a7e4d3-28e7-4cca-b729-de6502258dd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 13:56:55.258531 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.258379 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-token" (OuterVolumeSpecName: "istio-token") pod "38a7e4d3-28e7-4cca-b729-de6502258dd8" (UID: "38a7e4d3-28e7-4cca-b729-de6502258dd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:56:55.357343 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.357311 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-podinfo\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:55.357343 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.357339 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-envoy\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:55.357343 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.357349 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-credential-socket\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:55.357772 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.357358 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38a7e4d3-28e7-4cca-b729-de6502258dd8-istio-token\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:55.357772 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.357366 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs26h\" (UniqueName: \"kubernetes.io/projected/38a7e4d3-28e7-4cca-b729-de6502258dd8-kube-api-access-cs26h\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:55.357772 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.357375 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a7e4d3-28e7-4cca-b729-de6502258dd8-workload-socket\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:56:55.937475 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:55.937442 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:56.102586 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:56.102561 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29" Sep 16 13:56:56.125187 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:56.125163 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29"] Sep 16 13:56:56.129647 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:56.129596 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-bdd04e-openshift-default-85b89d85c8-zks29"] Sep 16 13:56:56.355104 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:56:56.355071 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38a7e4d3-28e7-4cca-b729-de6502258dd8" path="/var/lib/kubelet/pods/38a7e4d3-28e7-4cca-b729-de6502258dd8/volumes" Sep 16 13:57:00.162233 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:00.162203 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:00.272384 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:00.272348 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4bb29c-2z5jt"] Sep 16 13:57:00.272749 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:00.272736 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38a7e4d3-28e7-4cca-b729-de6502258dd8" containerName="istio-proxy" Sep 16 13:57:00.272816 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:00.272751 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="38a7e4d3-28e7-4cca-b729-de6502258dd8" containerName="istio-proxy" Sep 16 13:57:00.272816 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:00.272766 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38a7e4d3-28e7-4cca-b729-de6502258dd8" containerName="istio-proxy" Sep 16 13:57:00.272816 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:00.272771 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="38a7e4d3-28e7-4cca-b729-de6502258dd8" containerName="istio-proxy" Sep 16 13:57:00.272816 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:00.272779 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="953a2440-d7fc-4825-b31c-b2ed5720a0cd" containerName="revoke-keys" Sep 16 13:57:00.272816 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:00.272784 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="953a2440-d7fc-4825-b31c-b2ed5720a0cd" containerName="revoke-keys" Sep 16 13:57:00.272973 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:00.272850 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="38a7e4d3-28e7-4cca-b729-de6502258dd8" containerName="istio-proxy" Sep 16 13:57:00.272973 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:00.272858 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="953a2440-d7fc-4825-b31c-b2ed5720a0cd" containerName="revoke-keys" Sep 16 13:57:00.277359 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:00.277340 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4bb29c-2z5jt" Sep 16 13:57:00.280277 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:00.280251 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-thk5m\"" Sep 16 13:57:00.280462 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:00.280424 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:57:00.284678 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:00.284653 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4bb29c-2z5jt"] Sep 16 13:57:00.416668 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:00.416542 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/748d0131-3399-434d-a761-9fb0b8532e13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4bb29c-2z5jt\" (UID: \"748d0131-3399-434d-a761-9fb0b8532e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4bb29c-2z5jt" Sep 16 13:57:00.517561 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:00.517521 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/748d0131-3399-434d-a761-9fb0b8532e13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4bb29c-2z5jt\" (UID: \"748d0131-3399-434d-a761-9fb0b8532e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4bb29c-2z5jt" Sep 16 13:57:00.518156 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:00.518135 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/748d0131-3399-434d-a761-9fb0b8532e13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4bb29c-2z5jt\" (UID: \"748d0131-3399-434d-a761-9fb0b8532e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4bb29c-2z5jt" Sep 16 13:57:00.590057 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:00.590024 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4bb29c-2z5jt" Sep 16 13:57:00.867167 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:00.867138 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4bb29c-2z5jt"] Sep 16 13:57:00.868743 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:57:00.868716 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod748d0131_3399_434d_a761_9fb0b8532e13.slice/crio-ec2e99dd9d28d71463f9c1ffa1c02b7efb67eb3c3d5b08ebdd9986d09dd46622 WatchSource:0}: Error finding container ec2e99dd9d28d71463f9c1ffa1c02b7efb67eb3c3d5b08ebdd9986d09dd46622: Status 404 returned error can't find the container with id ec2e99dd9d28d71463f9c1ffa1c02b7efb67eb3c3d5b08ebdd9986d09dd46622 Sep 16 13:57:01.114676 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:01.114643 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:01.131433 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:01.131359 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4bb29c-2z5jt" event={"ID":"748d0131-3399-434d-a761-9fb0b8532e13","Type":"ContainerStarted","Data":"ec2e99dd9d28d71463f9c1ffa1c02b7efb67eb3c3d5b08ebdd9986d09dd46622"} Sep 16 13:57:01.820839 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:01.820793 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:02.138229 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:02.138198 2580 generic.go:356] "Generic (PLEG): container finished" podID="748d0131-3399-434d-a761-9fb0b8532e13" containerID="5393490eb9333667fe41d965163230d74c324f39638f737fdfa7d7012039e1d6" exitCode=0 Sep 16 13:57:02.138447 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:02.138281 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4bb29c-2z5jt" event={"ID":"748d0131-3399-434d-a761-9fb0b8532e13","Type":"ContainerDied","Data":"5393490eb9333667fe41d965163230d74c324f39638f737fdfa7d7012039e1d6"} Sep 16 13:57:02.498922 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:02.498880 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:02.996673 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:02.996633 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:57:03.278153 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:03.278125 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4bb29c-2z5jt" Sep 16 13:57:03.347691 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:03.347655 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/748d0131-3399-434d-a761-9fb0b8532e13-maas-api-service-ca\") pod \"748d0131-3399-434d-a761-9fb0b8532e13\" (UID: \"748d0131-3399-434d-a761-9fb0b8532e13\") " Sep 16 13:57:03.348003 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:03.347983 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/748d0131-3399-434d-a761-9fb0b8532e13-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "748d0131-3399-434d-a761-9fb0b8532e13" (UID: "748d0131-3399-434d-a761-9fb0b8532e13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:57:03.361747 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:03.361720 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:03.448890 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:03.448847 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/748d0131-3399-434d-a761-9fb0b8532e13-maas-api-service-ca\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:04.147875 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:04.147819 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:04.149047 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:04.149022 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4bb29c-2z5jt" Sep 16 13:57:04.149178 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:04.149054 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4bb29c-2z5jt" event={"ID":"748d0131-3399-434d-a761-9fb0b8532e13","Type":"ContainerDied","Data":"ec2e99dd9d28d71463f9c1ffa1c02b7efb67eb3c3d5b08ebdd9986d09dd46622"} Sep 16 13:57:04.149178 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:04.149119 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec2e99dd9d28d71463f9c1ffa1c02b7efb67eb3c3d5b08ebdd9986d09dd46622" Sep 16 13:57:05.142866 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:05.142832 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8"] Sep 16 13:57:05.148963 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:05.148935 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt"] Sep 16 13:57:05.166517 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:05.166467 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25"] Sep 16 13:57:05.181681 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:05.181651 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88"] Sep 16 13:57:05.224372 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:05.224342 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:05.287119 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:05.287088 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:10.446322 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:10.446286 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:11.031426 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:11.031396 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:11.108865 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:11.108836 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:11.179988 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:11.179955 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:11.249127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:11.249093 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59a5a8-w9hgh"] Sep 16 13:57:11.252820 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:11.252792 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59a5a8-w9hgh"] Sep 16 13:57:11.256178 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:11.256153 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:11.316001 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:11.315907 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:11.603065 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:11.602974 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:11.745072 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:11.745023 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:12.355666 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:12.355594 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="953a2440-d7fc-4825-b31c-b2ed5720a0cd" path="/var/lib/kubelet/pods/953a2440-d7fc-4825-b31c-b2ed5720a0cd/volumes" Sep 16 13:57:12.499331 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:12.499293 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:12.581398 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:12.581363 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:12.655974 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:12.655881 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:12.763775 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:12.763742 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:12.853807 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:12.853777 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:12.889249 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:12.889206 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4bb29c-2z5jt"] Sep 16 13:57:12.895416 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:12.895385 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4bb29c-2z5jt"] Sep 16 13:57:12.989784 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:12.989755 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:13.275095 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:13.274967 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:13.958233 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:13.958201 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:14.355916 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:14.355881 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="748d0131-3399-434d-a761-9fb0b8532e13" path="/var/lib/kubelet/pods/748d0131-3399-434d-a761-9fb0b8532e13/volumes" Sep 16 13:57:14.690958 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:14.690849 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:15.602984 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:15.602951 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:16.339381 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:16.339347 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:17.225536 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:17.225503 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:17.661361 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:17.661280 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:17.764480 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:17.764450 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:17.914671 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:17.914559 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l"] Sep 16 13:57:17.914837 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:17.914814 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" podUID="ed40eb9d-9f9a-496c-b298-ae975d3090a5" containerName="istio-proxy" containerID="cri-o://f9531c4291ce43f9edbbf7c32ac6e0e1a2f651b776decb7de2881b2f51aa2ec1" gracePeriod=30 Sep 16 13:57:17.937240 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:17.937213 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:19.042714 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:19.042668 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:19.798738 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:19.798699 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:19.962402 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:19.962372 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:20.064377 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.064287 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:20.163022 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.162983 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" podUID="a288cad1-3cb3-4953-ba84-471a45108ded" containerName="main" containerID="cri-o://d00a7a979cc5a21ff57214258cd699fcf5efcc6c191f2f9ac4688903abf5e5e6" gracePeriod=45 Sep 16 13:57:20.174633 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.174577 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" podUID="1e443f63-4448-4555-8e7f-7994b79dc069" containerName="main" containerID="cri-o://d4f8d9d61d30783dc881d4641514123d24c26cb0d2274c8b5f12bc1b6b1b86d0" gracePeriod=45 Sep 16 13:57:20.185820 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.185789 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" podUID="9fc23d93-10d4-4a71-ac2d-f9d881f008e3" containerName="main" containerID="cri-o://3d4a5bd04bb33281e5dd8ad6b7119461cfeea33c667337c67b2419c1039ad104" gracePeriod=45 Sep 16 13:57:20.201159 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.201107 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" podUID="3b14e93f-fcba-4664-bbd9-98475f21b719" containerName="main" containerID="cri-o://e5ff4bae40f58a128ad9497c9eb71439c040c7dabc1aec5a9bd0e138c8fde4fa" gracePeriod=45 Sep 16 13:57:20.244369 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.244342 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:20.460952 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.460906 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:57:20.510332 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.510278 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-tmp-dir\") pod \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " Sep 16 13:57:20.510479 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.510398 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-dshm\") pod \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " Sep 16 13:57:20.510479 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.510449 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-model-cache\") pod \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " Sep 16 13:57:20.510593 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.510504 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-home\") pod \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " Sep 16 13:57:20.510593 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.510549 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-tls-certs\") pod \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " Sep 16 13:57:20.510593 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.510585 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgvth\" (UniqueName: \"kubernetes.io/projected/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-kube-api-access-tgvth\") pod \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\" (UID: \"9fc23d93-10d4-4a71-ac2d-f9d881f008e3\") " Sep 16 13:57:20.511513 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.511472 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-home" (OuterVolumeSpecName: "home") pod "9fc23d93-10d4-4a71-ac2d-f9d881f008e3" (UID: "9fc23d93-10d4-4a71-ac2d-f9d881f008e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:20.511732 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.511696 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9fc23d93-10d4-4a71-ac2d-f9d881f008e3" (UID: "9fc23d93-10d4-4a71-ac2d-f9d881f008e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:20.512061 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.512038 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "9fc23d93-10d4-4a71-ac2d-f9d881f008e3" (UID: "9fc23d93-10d4-4a71-ac2d-f9d881f008e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:20.516059 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.516030 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-kube-api-access-tgvth" (OuterVolumeSpecName: "kube-api-access-tgvth") pod "9fc23d93-10d4-4a71-ac2d-f9d881f008e3" (UID: "9fc23d93-10d4-4a71-ac2d-f9d881f008e3"). InnerVolumeSpecName "kube-api-access-tgvth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:57:20.516441 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.516419 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:57:20.518219 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.518189 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9fc23d93-10d4-4a71-ac2d-f9d881f008e3" (UID: "9fc23d93-10d4-4a71-ac2d-f9d881f008e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:57:20.519065 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.519040 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-dshm" (OuterVolumeSpecName: "dshm") pod "9fc23d93-10d4-4a71-ac2d-f9d881f008e3" (UID: "9fc23d93-10d4-4a71-ac2d-f9d881f008e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:20.543748 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.543726 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:57:20.545787 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.545761 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:57:20.612226 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.612124 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-tmp-dir\") pod \"a288cad1-3cb3-4953-ba84-471a45108ded\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " Sep 16 13:57:20.612226 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.612172 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-dshm\") pod \"a288cad1-3cb3-4953-ba84-471a45108ded\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " Sep 16 13:57:20.612226 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.612196 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-model-cache\") pod \"a288cad1-3cb3-4953-ba84-471a45108ded\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " Sep 16 13:57:20.612226 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.612229 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-tmp-dir\") pod \"1e443f63-4448-4555-8e7f-7994b79dc069\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " Sep 16 13:57:20.612596 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.612261 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-model-cache\") pod \"3b14e93f-fcba-4664-bbd9-98475f21b719\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " Sep 16 13:57:20.612596 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.612315 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-dshm\") pod \"3b14e93f-fcba-4664-bbd9-98475f21b719\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " Sep 16 13:57:20.612596 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.612346 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-home\") pod \"1e443f63-4448-4555-8e7f-7994b79dc069\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " Sep 16 13:57:20.612596 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.612372 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnrvl\" (UniqueName: \"kubernetes.io/projected/3b14e93f-fcba-4664-bbd9-98475f21b719-kube-api-access-qnrvl\") pod \"3b14e93f-fcba-4664-bbd9-98475f21b719\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " Sep 16 13:57:20.612596 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.612411 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e443f63-4448-4555-8e7f-7994b79dc069-tls-certs\") pod \"1e443f63-4448-4555-8e7f-7994b79dc069\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " Sep 16 13:57:20.612596 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.612425 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a288cad1-3cb3-4953-ba84-471a45108ded" (UID: "a288cad1-3cb3-4953-ba84-471a45108ded"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:20.612596 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.612501 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-home\") pod \"a288cad1-3cb3-4953-ba84-471a45108ded\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " Sep 16 13:57:20.612596 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.612582 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1e443f63-4448-4555-8e7f-7994b79dc069" (UID: "1e443f63-4448-4555-8e7f-7994b79dc069"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:20.613066 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.612598 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kskt9\" (UniqueName: \"kubernetes.io/projected/a288cad1-3cb3-4953-ba84-471a45108ded-kube-api-access-kskt9\") pod \"a288cad1-3cb3-4953-ba84-471a45108ded\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " Sep 16 13:57:20.613066 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.612653 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-tmp-dir\") pod \"3b14e93f-fcba-4664-bbd9-98475f21b719\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " Sep 16 13:57:20.613066 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.612707 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a288cad1-3cb3-4953-ba84-471a45108ded-tls-certs\") pod \"a288cad1-3cb3-4953-ba84-471a45108ded\" (UID: \"a288cad1-3cb3-4953-ba84-471a45108ded\") " Sep 16 13:57:20.613066 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.612711 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b14e93f-fcba-4664-bbd9-98475f21b719" (UID: "3b14e93f-fcba-4664-bbd9-98475f21b719"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:20.613066 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.613048 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-model-cache" (OuterVolumeSpecName: "model-cache") pod "a288cad1-3cb3-4953-ba84-471a45108ded" (UID: "a288cad1-3cb3-4953-ba84-471a45108ded"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:20.613321 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.613243 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-home" (OuterVolumeSpecName: "home") pod "1e443f63-4448-4555-8e7f-7994b79dc069" (UID: "1e443f63-4448-4555-8e7f-7994b79dc069"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:20.614189 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.613866 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b14e93f-fcba-4664-bbd9-98475f21b719" (UID: "3b14e93f-fcba-4664-bbd9-98475f21b719"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:20.615745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.614371 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4b9q\" (UniqueName: \"kubernetes.io/projected/1e443f63-4448-4555-8e7f-7994b79dc069-kube-api-access-d4b9q\") pod \"1e443f63-4448-4555-8e7f-7994b79dc069\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " Sep 16 13:57:20.615745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.614440 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-home\") pod \"3b14e93f-fcba-4664-bbd9-98475f21b719\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " Sep 16 13:57:20.615745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.614479 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b14e93f-fcba-4664-bbd9-98475f21b719-tls-certs\") pod \"3b14e93f-fcba-4664-bbd9-98475f21b719\" (UID: \"3b14e93f-fcba-4664-bbd9-98475f21b719\") " Sep 16 13:57:20.615745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.614511 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-dshm\") pod \"1e443f63-4448-4555-8e7f-7994b79dc069\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " Sep 16 13:57:20.615745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.614541 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-model-cache\") pod \"1e443f63-4448-4555-8e7f-7994b79dc069\" (UID: \"1e443f63-4448-4555-8e7f-7994b79dc069\") " Sep 16 13:57:20.615745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.614938 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.615745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.614958 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.615745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.614974 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.615745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.614988 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgvth\" (UniqueName: \"kubernetes.io/projected/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-kube-api-access-tgvth\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.615745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.615003 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.615745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.615017 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.615745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.615029 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.615745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.615043 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.615745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.615056 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.615745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.615068 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.615745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.615080 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.615745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.615091 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc23d93-10d4-4a71-ac2d-f9d881f008e3-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.615745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.615439 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-home" (OuterVolumeSpecName: "home") pod "a288cad1-3cb3-4953-ba84-471a45108ded" (UID: "a288cad1-3cb3-4953-ba84-471a45108ded"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:20.615745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.615467 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-dshm" (OuterVolumeSpecName: "dshm") pod "3b14e93f-fcba-4664-bbd9-98475f21b719" (UID: "3b14e93f-fcba-4664-bbd9-98475f21b719"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:20.615745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.615489 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-dshm" (OuterVolumeSpecName: "dshm") pod "a288cad1-3cb3-4953-ba84-471a45108ded" (UID: "a288cad1-3cb3-4953-ba84-471a45108ded"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:20.615745 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.615506 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a288cad1-3cb3-4953-ba84-471a45108ded-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a288cad1-3cb3-4953-ba84-471a45108ded" (UID: "a288cad1-3cb3-4953-ba84-471a45108ded"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:57:20.617542 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.615694 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-home" (OuterVolumeSpecName: "home") pod "3b14e93f-fcba-4664-bbd9-98475f21b719" (UID: "3b14e93f-fcba-4664-bbd9-98475f21b719"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:20.617542 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.616366 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b14e93f-fcba-4664-bbd9-98475f21b719-kube-api-access-qnrvl" (OuterVolumeSpecName: "kube-api-access-qnrvl") pod "3b14e93f-fcba-4664-bbd9-98475f21b719" (UID: "3b14e93f-fcba-4664-bbd9-98475f21b719"). InnerVolumeSpecName "kube-api-access-qnrvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:57:20.617542 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.616699 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e443f63-4448-4555-8e7f-7994b79dc069-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e443f63-4448-4555-8e7f-7994b79dc069" (UID: "1e443f63-4448-4555-8e7f-7994b79dc069"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:57:20.617803 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.617752 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-model-cache" (OuterVolumeSpecName: "model-cache") pod "1e443f63-4448-4555-8e7f-7994b79dc069" (UID: "1e443f63-4448-4555-8e7f-7994b79dc069"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:20.617875 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.617850 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e443f63-4448-4555-8e7f-7994b79dc069-kube-api-access-d4b9q" (OuterVolumeSpecName: "kube-api-access-d4b9q") pod "1e443f63-4448-4555-8e7f-7994b79dc069" (UID: "1e443f63-4448-4555-8e7f-7994b79dc069"). InnerVolumeSpecName "kube-api-access-d4b9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:57:20.618196 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.618082 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-dshm" (OuterVolumeSpecName: "dshm") pod "1e443f63-4448-4555-8e7f-7994b79dc069" (UID: "1e443f63-4448-4555-8e7f-7994b79dc069"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:20.618196 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.618154 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b14e93f-fcba-4664-bbd9-98475f21b719-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b14e93f-fcba-4664-bbd9-98475f21b719" (UID: "3b14e93f-fcba-4664-bbd9-98475f21b719"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:57:20.618383 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.618239 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a288cad1-3cb3-4953-ba84-471a45108ded-kube-api-access-kskt9" (OuterVolumeSpecName: "kube-api-access-kskt9") pod "a288cad1-3cb3-4953-ba84-471a45108ded" (UID: "a288cad1-3cb3-4953-ba84-471a45108ded"). InnerVolumeSpecName "kube-api-access-kskt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:57:20.689428 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.689397 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:20.715968 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.715877 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b14e93f-fcba-4664-bbd9-98475f21b719-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.715968 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.715911 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.715968 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.715926 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e443f63-4448-4555-8e7f-7994b79dc069-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.715968 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.715940 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.715968 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.715951 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.715968 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.715963 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnrvl\" (UniqueName: \"kubernetes.io/projected/3b14e93f-fcba-4664-bbd9-98475f21b719-kube-api-access-qnrvl\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.715968 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.715976 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e443f63-4448-4555-8e7f-7994b79dc069-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.716456 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.715989 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a288cad1-3cb3-4953-ba84-471a45108ded-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.716456 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.716002 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kskt9\" (UniqueName: \"kubernetes.io/projected/a288cad1-3cb3-4953-ba84-471a45108ded-kube-api-access-kskt9\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.716456 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.716015 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a288cad1-3cb3-4953-ba84-471a45108ded-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.716456 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.716028 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4b9q\" (UniqueName: \"kubernetes.io/projected/1e443f63-4448-4555-8e7f-7994b79dc069-kube-api-access-d4b9q\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:20.716456 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:20.716040 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b14e93f-fcba-4664-bbd9-98475f21b719-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:21.237751 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.237712 2580 generic.go:356] "Generic (PLEG): container finished" podID="1e443f63-4448-4555-8e7f-7994b79dc069" containerID="d4f8d9d61d30783dc881d4641514123d24c26cb0d2274c8b5f12bc1b6b1b86d0" exitCode=0 Sep 16 13:57:21.238173 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.237758 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" Sep 16 13:57:21.238173 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.237803 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" event={"ID":"1e443f63-4448-4555-8e7f-7994b79dc069","Type":"ContainerDied","Data":"d4f8d9d61d30783dc881d4641514123d24c26cb0d2274c8b5f12bc1b6b1b86d0"} Sep 16 13:57:21.238173 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.237857 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt" event={"ID":"1e443f63-4448-4555-8e7f-7994b79dc069","Type":"ContainerDied","Data":"71b62219f21d60c1f21036c0cf891c6d0facda2531b952b3243cf6b956f1e11e"} Sep 16 13:57:21.238173 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.237883 2580 scope.go:117] "RemoveContainer" containerID="d4f8d9d61d30783dc881d4641514123d24c26cb0d2274c8b5f12bc1b6b1b86d0" Sep 16 13:57:21.239347 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.239324 2580 generic.go:356] "Generic (PLEG): container finished" podID="a288cad1-3cb3-4953-ba84-471a45108ded" containerID="d00a7a979cc5a21ff57214258cd699fcf5efcc6c191f2f9ac4688903abf5e5e6" exitCode=0 Sep 16 13:57:21.239402 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.239378 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" Sep 16 13:57:21.239448 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.239414 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" event={"ID":"a288cad1-3cb3-4953-ba84-471a45108ded","Type":"ContainerDied","Data":"d00a7a979cc5a21ff57214258cd699fcf5efcc6c191f2f9ac4688903abf5e5e6"} Sep 16 13:57:21.239501 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.239446 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8" event={"ID":"a288cad1-3cb3-4953-ba84-471a45108ded","Type":"ContainerDied","Data":"93c91ca8db9b90db94edd015661407d39764fc656aa8d38a4b0fef450520d114"} Sep 16 13:57:21.241148 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.241120 2580 generic.go:356] "Generic (PLEG): container finished" podID="3b14e93f-fcba-4664-bbd9-98475f21b719" containerID="e5ff4bae40f58a128ad9497c9eb71439c040c7dabc1aec5a9bd0e138c8fde4fa" exitCode=0 Sep 16 13:57:21.241297 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.241143 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" event={"ID":"3b14e93f-fcba-4664-bbd9-98475f21b719","Type":"ContainerDied","Data":"e5ff4bae40f58a128ad9497c9eb71439c040c7dabc1aec5a9bd0e138c8fde4fa"} Sep 16 13:57:21.241297 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.241184 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" Sep 16 13:57:21.241297 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.241199 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88" event={"ID":"3b14e93f-fcba-4664-bbd9-98475f21b719","Type":"ContainerDied","Data":"29e58bd2ec1b366cc02b9ff36d1aa4fd14fe1e108a923a672216ab72557c263a"} Sep 16 13:57:21.243416 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.243395 2580 generic.go:356] "Generic (PLEG): container finished" podID="9fc23d93-10d4-4a71-ac2d-f9d881f008e3" containerID="3d4a5bd04bb33281e5dd8ad6b7119461cfeea33c667337c67b2419c1039ad104" exitCode=0 Sep 16 13:57:21.243594 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.243436 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" Sep 16 13:57:21.243594 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.243480 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" event={"ID":"9fc23d93-10d4-4a71-ac2d-f9d881f008e3","Type":"ContainerDied","Data":"3d4a5bd04bb33281e5dd8ad6b7119461cfeea33c667337c67b2419c1039ad104"} Sep 16 13:57:21.243594 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.243518 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25" event={"ID":"9fc23d93-10d4-4a71-ac2d-f9d881f008e3","Type":"ContainerDied","Data":"1b43e365fbeca3f8a0fb33d2bb3ca10d38123d51c58223c5ab94ed4f5c7d83cb"} Sep 16 13:57:21.249044 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.249027 2580 scope.go:117] "RemoveContainer" containerID="d4f8d9d61d30783dc881d4641514123d24c26cb0d2274c8b5f12bc1b6b1b86d0" Sep 16 13:57:21.249350 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:57:21.249327 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4f8d9d61d30783dc881d4641514123d24c26cb0d2274c8b5f12bc1b6b1b86d0\": container with ID starting with d4f8d9d61d30783dc881d4641514123d24c26cb0d2274c8b5f12bc1b6b1b86d0 not found: ID does not exist" containerID="d4f8d9d61d30783dc881d4641514123d24c26cb0d2274c8b5f12bc1b6b1b86d0" Sep 16 13:57:21.249452 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.249359 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f8d9d61d30783dc881d4641514123d24c26cb0d2274c8b5f12bc1b6b1b86d0"} err="failed to get container status \"d4f8d9d61d30783dc881d4641514123d24c26cb0d2274c8b5f12bc1b6b1b86d0\": rpc error: code = NotFound desc = could not find container \"d4f8d9d61d30783dc881d4641514123d24c26cb0d2274c8b5f12bc1b6b1b86d0\": container with ID starting with d4f8d9d61d30783dc881d4641514123d24c26cb0d2274c8b5f12bc1b6b1b86d0 not found: ID does not exist" Sep 16 13:57:21.249452 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.249381 2580 scope.go:117] "RemoveContainer" containerID="d00a7a979cc5a21ff57214258cd699fcf5efcc6c191f2f9ac4688903abf5e5e6" Sep 16 13:57:21.258079 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.258060 2580 scope.go:117] "RemoveContainer" containerID="d00a7a979cc5a21ff57214258cd699fcf5efcc6c191f2f9ac4688903abf5e5e6" Sep 16 13:57:21.258345 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:57:21.258327 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d00a7a979cc5a21ff57214258cd699fcf5efcc6c191f2f9ac4688903abf5e5e6\": container with ID starting with d00a7a979cc5a21ff57214258cd699fcf5efcc6c191f2f9ac4688903abf5e5e6 not found: ID does not exist" containerID="d00a7a979cc5a21ff57214258cd699fcf5efcc6c191f2f9ac4688903abf5e5e6" Sep 16 13:57:21.258416 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.258350 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d00a7a979cc5a21ff57214258cd699fcf5efcc6c191f2f9ac4688903abf5e5e6"} err="failed to get container status \"d00a7a979cc5a21ff57214258cd699fcf5efcc6c191f2f9ac4688903abf5e5e6\": rpc error: code = NotFound desc = could not find container \"d00a7a979cc5a21ff57214258cd699fcf5efcc6c191f2f9ac4688903abf5e5e6\": container with ID starting with d00a7a979cc5a21ff57214258cd699fcf5efcc6c191f2f9ac4688903abf5e5e6 not found: ID does not exist" Sep 16 13:57:21.258416 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.258365 2580 scope.go:117] "RemoveContainer" containerID="e5ff4bae40f58a128ad9497c9eb71439c040c7dabc1aec5a9bd0e138c8fde4fa" Sep 16 13:57:21.266977 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.266951 2580 scope.go:117] "RemoveContainer" containerID="e5ff4bae40f58a128ad9497c9eb71439c040c7dabc1aec5a9bd0e138c8fde4fa" Sep 16 13:57:21.267274 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:57:21.267251 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5ff4bae40f58a128ad9497c9eb71439c040c7dabc1aec5a9bd0e138c8fde4fa\": container with ID starting with e5ff4bae40f58a128ad9497c9eb71439c040c7dabc1aec5a9bd0e138c8fde4fa not found: ID does not exist" containerID="e5ff4bae40f58a128ad9497c9eb71439c040c7dabc1aec5a9bd0e138c8fde4fa" Sep 16 13:57:21.267362 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.267278 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt"] Sep 16 13:57:21.267362 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.267280 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5ff4bae40f58a128ad9497c9eb71439c040c7dabc1aec5a9bd0e138c8fde4fa"} err="failed to get container status \"e5ff4bae40f58a128ad9497c9eb71439c040c7dabc1aec5a9bd0e138c8fde4fa\": rpc error: code = NotFound desc = could not find container \"e5ff4bae40f58a128ad9497c9eb71439c040c7dabc1aec5a9bd0e138c8fde4fa\": container with ID starting with e5ff4bae40f58a128ad9497c9eb71439c040c7dabc1aec5a9bd0e138c8fde4fa not found: ID does not exist" Sep 16 13:57:21.267362 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.267334 2580 scope.go:117] "RemoveContainer" containerID="3d4a5bd04bb33281e5dd8ad6b7119461cfeea33c667337c67b2419c1039ad104" Sep 16 13:57:21.270360 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.270336 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/distinct-w4-4bb29c-kserve-5c5d78bb65-pk2nt"] Sep 16 13:57:21.275619 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.275588 2580 scope.go:117] "RemoveContainer" containerID="3d4a5bd04bb33281e5dd8ad6b7119461cfeea33c667337c67b2419c1039ad104" Sep 16 13:57:21.275894 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:57:21.275876 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d4a5bd04bb33281e5dd8ad6b7119461cfeea33c667337c67b2419c1039ad104\": container with ID starting with 3d4a5bd04bb33281e5dd8ad6b7119461cfeea33c667337c67b2419c1039ad104 not found: ID does not exist" containerID="3d4a5bd04bb33281e5dd8ad6b7119461cfeea33c667337c67b2419c1039ad104" Sep 16 13:57:21.275958 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.275905 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d4a5bd04bb33281e5dd8ad6b7119461cfeea33c667337c67b2419c1039ad104"} err="failed to get container status \"3d4a5bd04bb33281e5dd8ad6b7119461cfeea33c667337c67b2419c1039ad104\": rpc error: code = NotFound desc = could not find container \"3d4a5bd04bb33281e5dd8ad6b7119461cfeea33c667337c67b2419c1039ad104\": container with ID starting with 3d4a5bd04bb33281e5dd8ad6b7119461cfeea33c667337c67b2419c1039ad104 not found: ID does not exist" Sep 16 13:57:21.281224 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.281203 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88"] Sep 16 13:57:21.286653 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.286630 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/simulator-w4-4bb29c-kserve-5f79546d7-f2w88"] Sep 16 13:57:21.301360 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.301339 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8"] Sep 16 13:57:21.307342 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.307322 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/distinct-2-w4-4bb29c-kserve-8659d88d5d-zdpv8"] Sep 16 13:57:21.319503 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.319483 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25"] Sep 16 13:57:21.324204 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.324184 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4bb29c/premium-w4-4bb29c-kserve-7994c56ff4-74f25"] Sep 16 13:57:21.398293 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:21.398263 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:22.239016 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:22.238983 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:22.355233 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:22.355190 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e443f63-4448-4555-8e7f-7994b79dc069" path="/var/lib/kubelet/pods/1e443f63-4448-4555-8e7f-7994b79dc069/volumes" Sep 16 13:57:22.355734 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:22.355712 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b14e93f-fcba-4664-bbd9-98475f21b719" path="/var/lib/kubelet/pods/3b14e93f-fcba-4664-bbd9-98475f21b719/volumes" Sep 16 13:57:22.356155 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:22.356135 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fc23d93-10d4-4a71-ac2d-f9d881f008e3" path="/var/lib/kubelet/pods/9fc23d93-10d4-4a71-ac2d-f9d881f008e3/volumes" Sep 16 13:57:22.356849 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:22.356829 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a288cad1-3cb3-4953-ba84-471a45108ded" path="/var/lib/kubelet/pods/a288cad1-3cb3-4953-ba84-471a45108ded/volumes" Sep 16 13:57:23.174291 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.174268 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:57:23.240801 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.240763 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-envoy\") pod \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " Sep 16 13:57:23.241231 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.240838 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-data\") pod \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " Sep 16 13:57:23.241231 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.240863 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-credential-socket\") pod \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " Sep 16 13:57:23.241231 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.240885 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istiod-ca-cert\") pod \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " Sep 16 13:57:23.241231 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.240903 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jfjk\" (UniqueName: \"kubernetes.io/projected/ed40eb9d-9f9a-496c-b298-ae975d3090a5-kube-api-access-8jfjk\") pod \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " Sep 16 13:57:23.241231 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.240932 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-podinfo\") pod \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " Sep 16 13:57:23.241231 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.240991 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-token\") pod \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " Sep 16 13:57:23.241231 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.241039 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-workload-certs\") pod \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " Sep 16 13:57:23.241231 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.241102 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-workload-socket\") pod \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\" (UID: \"ed40eb9d-9f9a-496c-b298-ae975d3090a5\") " Sep 16 13:57:23.241231 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.241156 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-data" (OuterVolumeSpecName: "istio-data") pod "ed40eb9d-9f9a-496c-b298-ae975d3090a5" (UID: "ed40eb9d-9f9a-496c-b298-ae975d3090a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:23.241231 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.241188 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ed40eb9d-9f9a-496c-b298-ae975d3090a5" (UID: "ed40eb9d-9f9a-496c-b298-ae975d3090a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:23.241987 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.241394 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ed40eb9d-9f9a-496c-b298-ae975d3090a5" (UID: "ed40eb9d-9f9a-496c-b298-ae975d3090a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:23.241987 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.241407 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ed40eb9d-9f9a-496c-b298-ae975d3090a5" (UID: "ed40eb9d-9f9a-496c-b298-ae975d3090a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:23.241987 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.241512 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-workload-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:23.241987 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.241533 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-workload-socket\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:23.241987 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.241546 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-data\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:23.241987 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.241559 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-credential-socket\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:23.241987 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.241771 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ed40eb9d-9f9a-496c-b298-ae975d3090a5" (UID: "ed40eb9d-9f9a-496c-b298-ae975d3090a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:57:23.243237 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.243209 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ed40eb9d-9f9a-496c-b298-ae975d3090a5" (UID: "ed40eb9d-9f9a-496c-b298-ae975d3090a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:23.243350 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.243334 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed40eb9d-9f9a-496c-b298-ae975d3090a5-kube-api-access-8jfjk" (OuterVolumeSpecName: "kube-api-access-8jfjk") pod "ed40eb9d-9f9a-496c-b298-ae975d3090a5" (UID: "ed40eb9d-9f9a-496c-b298-ae975d3090a5"). InnerVolumeSpecName "kube-api-access-8jfjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:57:23.243391 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.243369 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-token" (OuterVolumeSpecName: "istio-token") pod "ed40eb9d-9f9a-496c-b298-ae975d3090a5" (UID: "ed40eb9d-9f9a-496c-b298-ae975d3090a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:57:23.243391 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.243375 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ed40eb9d-9f9a-496c-b298-ae975d3090a5" (UID: "ed40eb9d-9f9a-496c-b298-ae975d3090a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 13:57:23.260010 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.259983 2580 generic.go:356] "Generic (PLEG): container finished" podID="ed40eb9d-9f9a-496c-b298-ae975d3090a5" containerID="f9531c4291ce43f9edbbf7c32ac6e0e1a2f651b776decb7de2881b2f51aa2ec1" exitCode=0 Sep 16 13:57:23.260150 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.260050 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" Sep 16 13:57:23.260150 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.260067 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" event={"ID":"ed40eb9d-9f9a-496c-b298-ae975d3090a5","Type":"ContainerDied","Data":"f9531c4291ce43f9edbbf7c32ac6e0e1a2f651b776decb7de2881b2f51aa2ec1"} Sep 16 13:57:23.260246 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.260150 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l" event={"ID":"ed40eb9d-9f9a-496c-b298-ae975d3090a5","Type":"ContainerDied","Data":"f50cbf1ed563539233dd763efd66298a39d32cb7d667d4db4c6c2e36e0a67d76"} Sep 16 13:57:23.260246 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.260173 2580 scope.go:117] "RemoveContainer" containerID="f9531c4291ce43f9edbbf7c32ac6e0e1a2f651b776decb7de2881b2f51aa2ec1" Sep 16 13:57:23.269206 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.269191 2580 scope.go:117] "RemoveContainer" containerID="50a9f50a570c6fa290b153cd0c34522ae6e887423c49dea5f813539db515b5ea" Sep 16 13:57:23.277113 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.277095 2580 scope.go:117] "RemoveContainer" containerID="f9531c4291ce43f9edbbf7c32ac6e0e1a2f651b776decb7de2881b2f51aa2ec1" Sep 16 13:57:23.277384 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:57:23.277366 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9531c4291ce43f9edbbf7c32ac6e0e1a2f651b776decb7de2881b2f51aa2ec1\": container with ID starting with f9531c4291ce43f9edbbf7c32ac6e0e1a2f651b776decb7de2881b2f51aa2ec1 not found: ID does not exist" containerID="f9531c4291ce43f9edbbf7c32ac6e0e1a2f651b776decb7de2881b2f51aa2ec1" Sep 16 13:57:23.277433 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.277393 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9531c4291ce43f9edbbf7c32ac6e0e1a2f651b776decb7de2881b2f51aa2ec1"} err="failed to get container status \"f9531c4291ce43f9edbbf7c32ac6e0e1a2f651b776decb7de2881b2f51aa2ec1\": rpc error: code = NotFound desc = could not find container \"f9531c4291ce43f9edbbf7c32ac6e0e1a2f651b776decb7de2881b2f51aa2ec1\": container with ID starting with f9531c4291ce43f9edbbf7c32ac6e0e1a2f651b776decb7de2881b2f51aa2ec1 not found: ID does not exist" Sep 16 13:57:23.277433 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.277409 2580 scope.go:117] "RemoveContainer" containerID="50a9f50a570c6fa290b153cd0c34522ae6e887423c49dea5f813539db515b5ea" Sep 16 13:57:23.277687 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:57:23.277670 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50a9f50a570c6fa290b153cd0c34522ae6e887423c49dea5f813539db515b5ea\": container with ID starting with 50a9f50a570c6fa290b153cd0c34522ae6e887423c49dea5f813539db515b5ea not found: ID does not exist" containerID="50a9f50a570c6fa290b153cd0c34522ae6e887423c49dea5f813539db515b5ea" Sep 16 13:57:23.277736 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.277692 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a9f50a570c6fa290b153cd0c34522ae6e887423c49dea5f813539db515b5ea"} err="failed to get container status \"50a9f50a570c6fa290b153cd0c34522ae6e887423c49dea5f813539db515b5ea\": rpc error: code = NotFound desc = could not find container \"50a9f50a570c6fa290b153cd0c34522ae6e887423c49dea5f813539db515b5ea\": container with ID starting with 50a9f50a570c6fa290b153cd0c34522ae6e887423c49dea5f813539db515b5ea not found: ID does not exist" Sep 16 13:57:23.282701 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.282680 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l"] Sep 16 13:57:23.286991 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.286969 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-59a5a8-openshift-default-56f7c74c7b-p942l"] Sep 16 13:57:23.342763 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.342732 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-envoy\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:23.342763 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.342759 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istiod-ca-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:23.342763 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.342768 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jfjk\" (UniqueName: \"kubernetes.io/projected/ed40eb9d-9f9a-496c-b298-ae975d3090a5-kube-api-access-8jfjk\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:23.342980 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.342779 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-podinfo\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:23.342980 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:23.342791 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed40eb9d-9f9a-496c-b298-ae975d3090a5-istio-token\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:24.356246 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:24.356214 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed40eb9d-9f9a-496c-b298-ae975d3090a5" path="/var/lib/kubelet/pods/ed40eb9d-9f9a-496c-b298-ae975d3090a5/volumes" Sep 16 13:57:28.748385 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:28.748352 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:28.849432 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:28.849390 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:28.944756 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:28.944713 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:29.008438 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:29.008355 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:29.392285 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:29.392200 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:29.480960 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:29.480922 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:30.459490 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:30.459462 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:31.108178 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:31.108145 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:57:31.261848 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:31.261816 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:32.049239 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:32.049162 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:35.897136 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:35.897095 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:35.986492 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:35.986455 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:36.208146 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.208027 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:36.562528 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.562490 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2afd5-8pqpp"] Sep 16 13:57:36.562861 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.562848 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="748d0131-3399-434d-a761-9fb0b8532e13" containerName="revoke-keys" Sep 16 13:57:36.562912 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.562864 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="748d0131-3399-434d-a761-9fb0b8532e13" containerName="revoke-keys" Sep 16 13:57:36.562912 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.562879 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed40eb9d-9f9a-496c-b298-ae975d3090a5" containerName="istio-proxy" Sep 16 13:57:36.562912 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.562884 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed40eb9d-9f9a-496c-b298-ae975d3090a5" containerName="istio-proxy" Sep 16 13:57:36.562912 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.562893 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b14e93f-fcba-4664-bbd9-98475f21b719" containerName="main" Sep 16 13:57:36.562912 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.562897 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b14e93f-fcba-4664-bbd9-98475f21b719" containerName="main" Sep 16 13:57:36.562912 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.562905 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed40eb9d-9f9a-496c-b298-ae975d3090a5" containerName="istio-proxy" Sep 16 13:57:36.562912 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.562910 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed40eb9d-9f9a-496c-b298-ae975d3090a5" containerName="istio-proxy" Sep 16 13:57:36.563127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.562919 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a288cad1-3cb3-4953-ba84-471a45108ded" containerName="main" Sep 16 13:57:36.563127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.562924 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a288cad1-3cb3-4953-ba84-471a45108ded" containerName="main" Sep 16 13:57:36.563127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.562931 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fc23d93-10d4-4a71-ac2d-f9d881f008e3" containerName="main" Sep 16 13:57:36.563127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.562936 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fc23d93-10d4-4a71-ac2d-f9d881f008e3" containerName="main" Sep 16 13:57:36.563127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.562944 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e443f63-4448-4555-8e7f-7994b79dc069" containerName="main" Sep 16 13:57:36.563127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.562949 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e443f63-4448-4555-8e7f-7994b79dc069" containerName="main" Sep 16 13:57:36.563127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.563003 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed40eb9d-9f9a-496c-b298-ae975d3090a5" containerName="istio-proxy" Sep 16 13:57:36.563127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.563011 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e443f63-4448-4555-8e7f-7994b79dc069" containerName="main" Sep 16 13:57:36.563127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.563017 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a288cad1-3cb3-4953-ba84-471a45108ded" containerName="main" Sep 16 13:57:36.563127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.563022 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b14e93f-fcba-4664-bbd9-98475f21b719" containerName="main" Sep 16 13:57:36.563127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.563028 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fc23d93-10d4-4a71-ac2d-f9d881f008e3" containerName="main" Sep 16 13:57:36.563127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.563034 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed40eb9d-9f9a-496c-b298-ae975d3090a5" containerName="istio-proxy" Sep 16 13:57:36.563127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.563040 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="38a7e4d3-28e7-4cca-b729-de6502258dd8" containerName="istio-proxy" Sep 16 13:57:36.563127 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.563047 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="748d0131-3399-434d-a761-9fb0b8532e13" containerName="revoke-keys" Sep 16 13:57:36.567527 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.567508 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2afd5-8pqpp" Sep 16 13:57:36.571310 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.571288 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-thk5m\"" Sep 16 13:57:36.577618 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.577580 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2afd5-8pqpp"] Sep 16 13:57:36.580920 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.580900 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:57:36.671233 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.671189 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e169ff6-9dd9-41b6-a7e6-bbc4508fa094-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f2afd5-8pqpp\" (UID: \"4e169ff6-9dd9-41b6-a7e6-bbc4508fa094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2afd5-8pqpp" Sep 16 13:57:36.771952 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.771914 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e169ff6-9dd9-41b6-a7e6-bbc4508fa094-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f2afd5-8pqpp\" (UID: \"4e169ff6-9dd9-41b6-a7e6-bbc4508fa094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2afd5-8pqpp" Sep 16 13:57:36.772544 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.772522 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e169ff6-9dd9-41b6-a7e6-bbc4508fa094-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f2afd5-8pqpp\" (UID: \"4e169ff6-9dd9-41b6-a7e6-bbc4508fa094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2afd5-8pqpp" Sep 16 13:57:36.877898 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.877805 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2afd5-8pqpp" Sep 16 13:57:36.899853 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:36.899826 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:37.163708 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:37.163680 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2afd5-8pqpp"] Sep 16 13:57:37.164886 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:57:37.164861 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e169ff6_9dd9_41b6_a7e6_bbc4508fa094.slice/crio-f166ac050aa599464e68c5b4d54bbfb3c5abe236250b0a6d8385163fe2fde75f WatchSource:0}: Error finding container f166ac050aa599464e68c5b4d54bbfb3c5abe236250b0a6d8385163fe2fde75f: Status 404 returned error can't find the container with id f166ac050aa599464e68c5b4d54bbfb3c5abe236250b0a6d8385163fe2fde75f Sep 16 13:57:37.330015 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:37.329984 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2afd5-8pqpp" event={"ID":"4e169ff6-9dd9-41b6-a7e6-bbc4508fa094","Type":"ContainerStarted","Data":"f166ac050aa599464e68c5b4d54bbfb3c5abe236250b0a6d8385163fe2fde75f"} Sep 16 13:57:37.740992 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:37.740956 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:38.336492 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:38.336455 2580 generic.go:356] "Generic (PLEG): container finished" podID="4e169ff6-9dd9-41b6-a7e6-bbc4508fa094" containerID="302206e4e9084f2ffd8fb5e5ee2b8bd2d79ff06747b0c1d5909464f8228e7756" exitCode=0 Sep 16 13:57:38.336933 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:38.336526 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2afd5-8pqpp" event={"ID":"4e169ff6-9dd9-41b6-a7e6-bbc4508fa094","Type":"ContainerDied","Data":"302206e4e9084f2ffd8fb5e5ee2b8bd2d79ff06747b0c1d5909464f8228e7756"} Sep 16 13:57:38.423595 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:38.423558 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:39.083340 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:39.083307 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:39.481918 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:39.481890 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2afd5-8pqpp" Sep 16 13:57:39.599513 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:39.599420 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e169ff6-9dd9-41b6-a7e6-bbc4508fa094-maas-api-service-ca\") pod \"4e169ff6-9dd9-41b6-a7e6-bbc4508fa094\" (UID: \"4e169ff6-9dd9-41b6-a7e6-bbc4508fa094\") " Sep 16 13:57:39.599891 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:39.599864 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e169ff6-9dd9-41b6-a7e6-bbc4508fa094-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e169ff6-9dd9-41b6-a7e6-bbc4508fa094" (UID: "4e169ff6-9dd9-41b6-a7e6-bbc4508fa094"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:57:39.700342 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:39.700304 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e169ff6-9dd9-41b6-a7e6-bbc4508fa094-maas-api-service-ca\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:57:40.038445 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:40.038413 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:40.347322 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:40.347232 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2afd5-8pqpp" Sep 16 13:57:40.347322 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:40.347265 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2afd5-8pqpp" event={"ID":"4e169ff6-9dd9-41b6-a7e6-bbc4508fa094","Type":"ContainerDied","Data":"f166ac050aa599464e68c5b4d54bbfb3c5abe236250b0a6d8385163fe2fde75f"} Sep 16 13:57:40.347506 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:40.347342 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f166ac050aa599464e68c5b4d54bbfb3c5abe236250b0a6d8385163fe2fde75f" Sep 16 13:57:41.664901 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:41.664869 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:41.765285 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:41.765253 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:43.056265 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:43.056235 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:47.245331 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:47.245297 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:47.349333 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:47.349301 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:57:47.404026 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:47.403946 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:47.505856 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:47.505767 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:47.611169 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:47.611129 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2afd5-8pqpp"] Sep 16 13:57:47.614477 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:47.614432 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:47.616058 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:47.616040 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f2afd5-8pqpp"] Sep 16 13:57:47.707349 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:47.707316 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:48.354402 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:48.354367 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e169ff6-9dd9-41b6-a7e6-bbc4508fa094" path="/var/lib/kubelet/pods/4e169ff6-9dd9-41b6-a7e6-bbc4508fa094/volumes" Sep 16 13:57:49.098202 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:49.098167 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:49.267709 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:49.267677 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:50.155266 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:50.155232 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:50.672775 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:50.672742 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:54.495498 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:54.495464 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:54.595961 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:54.595928 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:54.771644 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:54.771544 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8"] Sep 16 13:57:54.771838 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:54.771814 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" podUID="5d852849-d15b-4de8-9b0c-3d4925a61534" containerName="istio-proxy" containerID="cri-o://46d94259b341e1b013a2955eecf672b2f923a75ac17f4d941f5460fea7a7ade5" gracePeriod=30 Sep 16 13:57:54.836707 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:54.836660 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:57:59.055824 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:57:59.055789 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:58:00.022344 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.022320 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:58:00.092702 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.092615 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-podinfo\") pod \"5d852849-d15b-4de8-9b0c-3d4925a61534\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " Sep 16 13:58:00.092702 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.092664 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c6c2\" (UniqueName: \"kubernetes.io/projected/5d852849-d15b-4de8-9b0c-3d4925a61534-kube-api-access-6c6c2\") pod \"5d852849-d15b-4de8-9b0c-3d4925a61534\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " Sep 16 13:58:00.092702 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.092693 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-envoy\") pod \"5d852849-d15b-4de8-9b0c-3d4925a61534\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " Sep 16 13:58:00.093341 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.092715 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-data\") pod \"5d852849-d15b-4de8-9b0c-3d4925a61534\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " Sep 16 13:58:00.093341 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.092772 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-workload-socket\") pod \"5d852849-d15b-4de8-9b0c-3d4925a61534\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " Sep 16 13:58:00.093341 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.092812 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-token\") pod \"5d852849-d15b-4de8-9b0c-3d4925a61534\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " Sep 16 13:58:00.093341 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.092885 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d852849-d15b-4de8-9b0c-3d4925a61534-istiod-ca-cert\") pod \"5d852849-d15b-4de8-9b0c-3d4925a61534\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " Sep 16 13:58:00.093341 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.092915 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-workload-certs\") pod \"5d852849-d15b-4de8-9b0c-3d4925a61534\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " Sep 16 13:58:00.093341 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.092942 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-credential-socket\") pod \"5d852849-d15b-4de8-9b0c-3d4925a61534\" (UID: \"5d852849-d15b-4de8-9b0c-3d4925a61534\") " Sep 16 13:58:00.093341 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.093131 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-data" (OuterVolumeSpecName: "istio-data") pod "5d852849-d15b-4de8-9b0c-3d4925a61534" (UID: "5d852849-d15b-4de8-9b0c-3d4925a61534"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:58:00.093341 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.093327 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5d852849-d15b-4de8-9b0c-3d4925a61534" (UID: "5d852849-d15b-4de8-9b0c-3d4925a61534"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:58:00.093748 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.093366 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-data\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:58:00.093748 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.093462 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5d852849-d15b-4de8-9b0c-3d4925a61534" (UID: "5d852849-d15b-4de8-9b0c-3d4925a61534"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:58:00.093748 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.093688 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d852849-d15b-4de8-9b0c-3d4925a61534-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5d852849-d15b-4de8-9b0c-3d4925a61534" (UID: "5d852849-d15b-4de8-9b0c-3d4925a61534"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:58:00.093896 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.093874 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5d852849-d15b-4de8-9b0c-3d4925a61534" (UID: "5d852849-d15b-4de8-9b0c-3d4925a61534"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:58:00.095177 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.095146 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5d852849-d15b-4de8-9b0c-3d4925a61534" (UID: "5d852849-d15b-4de8-9b0c-3d4925a61534"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 13:58:00.095177 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.095159 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d852849-d15b-4de8-9b0c-3d4925a61534-kube-api-access-6c6c2" (OuterVolumeSpecName: "kube-api-access-6c6c2") pod "5d852849-d15b-4de8-9b0c-3d4925a61534" (UID: "5d852849-d15b-4de8-9b0c-3d4925a61534"). InnerVolumeSpecName "kube-api-access-6c6c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:58:00.095578 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.095559 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5d852849-d15b-4de8-9b0c-3d4925a61534" (UID: "5d852849-d15b-4de8-9b0c-3d4925a61534"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:58:00.095736 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.095715 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-token" (OuterVolumeSpecName: "istio-token") pod "5d852849-d15b-4de8-9b0c-3d4925a61534" (UID: "5d852849-d15b-4de8-9b0c-3d4925a61534"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:58:00.194510 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.194474 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-podinfo\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:58:00.194510 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.194504 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c6c2\" (UniqueName: \"kubernetes.io/projected/5d852849-d15b-4de8-9b0c-3d4925a61534-kube-api-access-6c6c2\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:58:00.194510 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.194514 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-envoy\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:58:00.194785 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.194522 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-workload-socket\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:58:00.194785 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.194530 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d852849-d15b-4de8-9b0c-3d4925a61534-istio-token\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:58:00.194785 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.194537 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d852849-d15b-4de8-9b0c-3d4925a61534-istiod-ca-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:58:00.194785 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.194545 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-workload-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:58:00.194785 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.194551 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d852849-d15b-4de8-9b0c-3d4925a61534-credential-socket\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:58:00.447633 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.447582 2580 generic.go:356] "Generic (PLEG): container finished" podID="5d852849-d15b-4de8-9b0c-3d4925a61534" containerID="46d94259b341e1b013a2955eecf672b2f923a75ac17f4d941f5460fea7a7ade5" exitCode=0 Sep 16 13:58:00.447828 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.447644 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" event={"ID":"5d852849-d15b-4de8-9b0c-3d4925a61534","Type":"ContainerDied","Data":"46d94259b341e1b013a2955eecf672b2f923a75ac17f4d941f5460fea7a7ade5"} Sep 16 13:58:00.447828 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.447677 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" Sep 16 13:58:00.447828 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.447724 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8" event={"ID":"5d852849-d15b-4de8-9b0c-3d4925a61534","Type":"ContainerDied","Data":"9b97f85b7b3dbfeade2a2955f26ee235b4345103e127c8e18eb92b40f05b2086"} Sep 16 13:58:00.447828 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.447755 2580 scope.go:117] "RemoveContainer" containerID="46d94259b341e1b013a2955eecf672b2f923a75ac17f4d941f5460fea7a7ade5" Sep 16 13:58:00.456527 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.456503 2580 scope.go:117] "RemoveContainer" containerID="0be99a08b3d3f1c82ae5edc57a015094407d848c14e0e3dd15f72559a09d51c0" Sep 16 13:58:00.464542 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.464512 2580 scope.go:117] "RemoveContainer" containerID="46d94259b341e1b013a2955eecf672b2f923a75ac17f4d941f5460fea7a7ade5" Sep 16 13:58:00.464847 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:58:00.464828 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46d94259b341e1b013a2955eecf672b2f923a75ac17f4d941f5460fea7a7ade5\": container with ID starting with 46d94259b341e1b013a2955eecf672b2f923a75ac17f4d941f5460fea7a7ade5 not found: ID does not exist" containerID="46d94259b341e1b013a2955eecf672b2f923a75ac17f4d941f5460fea7a7ade5" Sep 16 13:58:00.464925 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.464858 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d94259b341e1b013a2955eecf672b2f923a75ac17f4d941f5460fea7a7ade5"} err="failed to get container status \"46d94259b341e1b013a2955eecf672b2f923a75ac17f4d941f5460fea7a7ade5\": rpc error: code = NotFound desc = could not find container \"46d94259b341e1b013a2955eecf672b2f923a75ac17f4d941f5460fea7a7ade5\": container with ID starting with 46d94259b341e1b013a2955eecf672b2f923a75ac17f4d941f5460fea7a7ade5 not found: ID does not exist" Sep 16 13:58:00.464925 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.464882 2580 scope.go:117] "RemoveContainer" containerID="0be99a08b3d3f1c82ae5edc57a015094407d848c14e0e3dd15f72559a09d51c0" Sep 16 13:58:00.465168 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:58:00.465150 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0be99a08b3d3f1c82ae5edc57a015094407d848c14e0e3dd15f72559a09d51c0\": container with ID starting with 0be99a08b3d3f1c82ae5edc57a015094407d848c14e0e3dd15f72559a09d51c0 not found: ID does not exist" containerID="0be99a08b3d3f1c82ae5edc57a015094407d848c14e0e3dd15f72559a09d51c0" Sep 16 13:58:00.465208 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.465174 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0be99a08b3d3f1c82ae5edc57a015094407d848c14e0e3dd15f72559a09d51c0"} err="failed to get container status \"0be99a08b3d3f1c82ae5edc57a015094407d848c14e0e3dd15f72559a09d51c0\": rpc error: code = NotFound desc = could not find container \"0be99a08b3d3f1c82ae5edc57a015094407d848c14e0e3dd15f72559a09d51c0\": container with ID starting with 0be99a08b3d3f1c82ae5edc57a015094407d848c14e0e3dd15f72559a09d51c0 not found: ID does not exist" Sep 16 13:58:00.472834 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.472811 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8"] Sep 16 13:58:00.482295 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:00.482275 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2afd5-openshift-default-555797549b-j96v8"] Sep 16 13:58:02.356322 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:02.356276 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d852849-d15b-4de8-9b0c-3d4925a61534" path="/var/lib/kubelet/pods/5d852849-d15b-4de8-9b0c-3d4925a61534/volumes" Sep 16 13:58:10.443118 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:10.443083 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:58:11.383871 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:11.383833 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx"] Sep 16 13:58:12.252882 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:12.252847 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77"] Sep 16 13:58:12.352200 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:12.352156 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:12.442210 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:12.442183 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:18.610423 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:18.610381 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:58:26.402862 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.402821 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" podUID="39786424-c91c-427a-a41a-1e4758affc48" containerName="main" containerID="cri-o://cf1104d9d676d623876d18c2b7281f2c550e83178c328099218c1063540f004e" gracePeriod=45 Sep 16 13:58:26.574526 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.574498 2580 generic.go:356] "Generic (PLEG): container finished" podID="39786424-c91c-427a-a41a-1e4758affc48" containerID="cf1104d9d676d623876d18c2b7281f2c550e83178c328099218c1063540f004e" exitCode=0 Sep 16 13:58:26.574671 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.574566 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" event={"ID":"39786424-c91c-427a-a41a-1e4758affc48","Type":"ContainerDied","Data":"cf1104d9d676d623876d18c2b7281f2c550e83178c328099218c1063540f004e"} Sep 16 13:58:26.644414 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.644387 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:58:26.733844 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.731310 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9"] Sep 16 13:58:26.733844 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.732067 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d852849-d15b-4de8-9b0c-3d4925a61534" containerName="istio-proxy" Sep 16 13:58:26.733844 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.732087 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d852849-d15b-4de8-9b0c-3d4925a61534" containerName="istio-proxy" Sep 16 13:58:26.733844 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.732103 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e169ff6-9dd9-41b6-a7e6-bbc4508fa094" containerName="revoke-keys" Sep 16 13:58:26.733844 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.732112 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e169ff6-9dd9-41b6-a7e6-bbc4508fa094" containerName="revoke-keys" Sep 16 13:58:26.733844 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.732156 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39786424-c91c-427a-a41a-1e4758affc48" containerName="main" Sep 16 13:58:26.733844 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.732165 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="39786424-c91c-427a-a41a-1e4758affc48" containerName="main" Sep 16 13:58:26.733844 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.732309 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d852849-d15b-4de8-9b0c-3d4925a61534" containerName="istio-proxy" Sep 16 13:58:26.733844 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.732320 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d852849-d15b-4de8-9b0c-3d4925a61534" containerName="istio-proxy" Sep 16 13:58:26.733844 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.732335 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e169ff6-9dd9-41b6-a7e6-bbc4508fa094" containerName="revoke-keys" Sep 16 13:58:26.733844 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.732342 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="39786424-c91c-427a-a41a-1e4758affc48" containerName="main" Sep 16 13:58:26.735832 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.735807 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-home\") pod \"39786424-c91c-427a-a41a-1e4758affc48\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " Sep 16 13:58:26.735959 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.735855 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-model-cache\") pod \"39786424-c91c-427a-a41a-1e4758affc48\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " Sep 16 13:58:26.735959 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.735883 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvzqr\" (UniqueName: \"kubernetes.io/projected/39786424-c91c-427a-a41a-1e4758affc48-kube-api-access-hvzqr\") pod \"39786424-c91c-427a-a41a-1e4758affc48\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " Sep 16 13:58:26.736090 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.736071 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39786424-c91c-427a-a41a-1e4758affc48-tls-certs\") pod \"39786424-c91c-427a-a41a-1e4758affc48\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " Sep 16 13:58:26.736151 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.736134 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-tmp-dir\") pod \"39786424-c91c-427a-a41a-1e4758affc48\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " Sep 16 13:58:26.736209 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.736174 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-dshm\") pod \"39786424-c91c-427a-a41a-1e4758affc48\" (UID: \"39786424-c91c-427a-a41a-1e4758affc48\") " Sep 16 13:58:26.736262 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.736209 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-model-cache" (OuterVolumeSpecName: "model-cache") pod "39786424-c91c-427a-a41a-1e4758affc48" (UID: "39786424-c91c-427a-a41a-1e4758affc48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:58:26.736262 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.736225 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-home" (OuterVolumeSpecName: "home") pod "39786424-c91c-427a-a41a-1e4758affc48" (UID: "39786424-c91c-427a-a41a-1e4758affc48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:58:26.736424 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.736407 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:58:26.736489 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.736429 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:58:26.736555 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.736482 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "39786424-c91c-427a-a41a-1e4758affc48" (UID: "39786424-c91c-427a-a41a-1e4758affc48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:58:26.736555 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.736511 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:26.738562 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.738526 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-dshm" (OuterVolumeSpecName: "dshm") pod "39786424-c91c-427a-a41a-1e4758affc48" (UID: "39786424-c91c-427a-a41a-1e4758affc48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:58:26.738562 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.738527 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39786424-c91c-427a-a41a-1e4758affc48-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "39786424-c91c-427a-a41a-1e4758affc48" (UID: "39786424-c91c-427a-a41a-1e4758affc48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:58:26.738779 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.738591 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39786424-c91c-427a-a41a-1e4758affc48-kube-api-access-hvzqr" (OuterVolumeSpecName: "kube-api-access-hvzqr") pod "39786424-c91c-427a-a41a-1e4758affc48" (UID: "39786424-c91c-427a-a41a-1e4758affc48"). InnerVolumeSpecName "kube-api-access-hvzqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:58:26.752475 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.752455 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-973610\"/\"sub-test-model-973610-kserve-self-signed-certs\"" Sep 16 13:58:26.769517 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.769491 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9"] Sep 16 13:58:26.832856 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.832821 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:26.836911 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.836887 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-tmp-dir\") pod \"sub-test-model-973610-kserve-67f448679f-wmff9\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:26.837062 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.836940 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-tls-certs\") pod \"sub-test-model-973610-kserve-67f448679f-wmff9\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:26.837062 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.836966 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-home\") pod \"sub-test-model-973610-kserve-67f448679f-wmff9\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:26.837062 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.836991 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-dshm\") pod \"sub-test-model-973610-kserve-67f448679f-wmff9\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:26.837062 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.837032 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smk8k\" (UniqueName: \"kubernetes.io/projected/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-kube-api-access-smk8k\") pod \"sub-test-model-973610-kserve-67f448679f-wmff9\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:26.837222 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.837079 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-model-cache\") pod \"sub-test-model-973610-kserve-67f448679f-wmff9\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:26.837222 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.837118 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvzqr\" (UniqueName: \"kubernetes.io/projected/39786424-c91c-427a-a41a-1e4758affc48-kube-api-access-hvzqr\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:58:26.837222 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.837129 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39786424-c91c-427a-a41a-1e4758affc48-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:58:26.837222 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.837138 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:58:26.837222 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.837151 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39786424-c91c-427a-a41a-1e4758affc48-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:58:26.937707 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.937669 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-tmp-dir\") pod \"sub-test-model-973610-kserve-67f448679f-wmff9\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:26.937909 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.937744 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-tls-certs\") pod \"sub-test-model-973610-kserve-67f448679f-wmff9\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:26.937909 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.937781 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-home\") pod \"sub-test-model-973610-kserve-67f448679f-wmff9\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:26.937909 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.937799 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-dshm\") pod \"sub-test-model-973610-kserve-67f448679f-wmff9\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:26.938076 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.937973 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smk8k\" (UniqueName: \"kubernetes.io/projected/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-kube-api-access-smk8k\") pod \"sub-test-model-973610-kserve-67f448679f-wmff9\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:26.938076 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.938042 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-model-cache\") pod \"sub-test-model-973610-kserve-67f448679f-wmff9\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:26.938182 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.938165 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-tmp-dir\") pod \"sub-test-model-973610-kserve-67f448679f-wmff9\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:26.938235 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.938178 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-home\") pod \"sub-test-model-973610-kserve-67f448679f-wmff9\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:26.938329 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.938312 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-model-cache\") pod \"sub-test-model-973610-kserve-67f448679f-wmff9\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:26.940338 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.940315 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-dshm\") pod \"sub-test-model-973610-kserve-67f448679f-wmff9\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:26.940489 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.940472 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-tls-certs\") pod \"sub-test-model-973610-kserve-67f448679f-wmff9\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:26.949528 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:26.949505 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smk8k\" (UniqueName: \"kubernetes.io/projected/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-kube-api-access-smk8k\") pod \"sub-test-model-973610-kserve-67f448679f-wmff9\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:27.053488 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.053395 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:27.272048 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.271985 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" podUID="12148179-76e7-4a24-bc06-932f8b764b43" containerName="main" containerID="cri-o://5b89625eb7a716f4633e7fe2b520bc4f353c061246b8bf75750374098b88430c" gracePeriod=45 Sep 16 13:58:27.335508 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.335479 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9"] Sep 16 13:58:27.340640 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:58:27.340594 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26c3ca9b_22cc_430a_bcff_bfbf13bc4b43.slice/crio-5da2528cabcb88514736ba9267af83a7a67e2430f96a1717877ecc2878007456 WatchSource:0}: Error finding container 5da2528cabcb88514736ba9267af83a7a67e2430f96a1717877ecc2878007456: Status 404 returned error can't find the container with id 5da2528cabcb88514736ba9267af83a7a67e2430f96a1717877ecc2878007456 Sep 16 13:58:27.523065 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.523037 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:58:27.543906 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.543870 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12148179-76e7-4a24-bc06-932f8b764b43-tls-certs\") pod \"12148179-76e7-4a24-bc06-932f8b764b43\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " Sep 16 13:58:27.543906 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.543912 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-dshm\") pod \"12148179-76e7-4a24-bc06-932f8b764b43\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " Sep 16 13:58:27.544147 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.543934 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9j5m9\" (UniqueName: \"kubernetes.io/projected/12148179-76e7-4a24-bc06-932f8b764b43-kube-api-access-9j5m9\") pod \"12148179-76e7-4a24-bc06-932f8b764b43\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " Sep 16 13:58:27.544147 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.543976 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-model-cache\") pod \"12148179-76e7-4a24-bc06-932f8b764b43\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " Sep 16 13:58:27.544147 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.544023 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-tmp-dir\") pod \"12148179-76e7-4a24-bc06-932f8b764b43\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " Sep 16 13:58:27.544147 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.544083 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-home\") pod \"12148179-76e7-4a24-bc06-932f8b764b43\" (UID: \"12148179-76e7-4a24-bc06-932f8b764b43\") " Sep 16 13:58:27.544377 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.544265 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-model-cache" (OuterVolumeSpecName: "model-cache") pod "12148179-76e7-4a24-bc06-932f8b764b43" (UID: "12148179-76e7-4a24-bc06-932f8b764b43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:58:27.544377 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.544309 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12148179-76e7-4a24-bc06-932f8b764b43" (UID: "12148179-76e7-4a24-bc06-932f8b764b43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:58:27.544484 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.544460 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-home" (OuterVolumeSpecName: "home") pod "12148179-76e7-4a24-bc06-932f8b764b43" (UID: "12148179-76e7-4a24-bc06-932f8b764b43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:58:27.546462 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.546435 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-dshm" (OuterVolumeSpecName: "dshm") pod "12148179-76e7-4a24-bc06-932f8b764b43" (UID: "12148179-76e7-4a24-bc06-932f8b764b43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:58:27.546646 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.546475 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12148179-76e7-4a24-bc06-932f8b764b43-kube-api-access-9j5m9" (OuterVolumeSpecName: "kube-api-access-9j5m9") pod "12148179-76e7-4a24-bc06-932f8b764b43" (UID: "12148179-76e7-4a24-bc06-932f8b764b43"). InnerVolumeSpecName "kube-api-access-9j5m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:58:27.546646 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.546523 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12148179-76e7-4a24-bc06-932f8b764b43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12148179-76e7-4a24-bc06-932f8b764b43" (UID: "12148179-76e7-4a24-bc06-932f8b764b43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:58:27.580879 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.580836 2580 generic.go:356] "Generic (PLEG): container finished" podID="12148179-76e7-4a24-bc06-932f8b764b43" containerID="5b89625eb7a716f4633e7fe2b520bc4f353c061246b8bf75750374098b88430c" exitCode=0 Sep 16 13:58:27.580879 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.580876 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" Sep 16 13:58:27.581106 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.580925 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" event={"ID":"12148179-76e7-4a24-bc06-932f8b764b43","Type":"ContainerDied","Data":"5b89625eb7a716f4633e7fe2b520bc4f353c061246b8bf75750374098b88430c"} Sep 16 13:58:27.581106 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.580970 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77" event={"ID":"12148179-76e7-4a24-bc06-932f8b764b43","Type":"ContainerDied","Data":"0f408f1949ff37324f5a390aa9cdedbf860092f1a470d2ead8bb76f2d33e9793"} Sep 16 13:58:27.581106 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.580992 2580 scope.go:117] "RemoveContainer" containerID="5b89625eb7a716f4633e7fe2b520bc4f353c061246b8bf75750374098b88430c" Sep 16 13:58:27.582418 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.582397 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" event={"ID":"39786424-c91c-427a-a41a-1e4758affc48","Type":"ContainerDied","Data":"71d5f2845c8f35c10dba520bb0be4393dc1b96a9eeaa24ae155243181cbdacb5"} Sep 16 13:58:27.582514 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.582399 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx" Sep 16 13:58:27.584245 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.584217 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" event={"ID":"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43","Type":"ContainerStarted","Data":"2b94774e47fb8b9ede4550244b4badd2a89b47663bb7e51dec6e5eecc1a86229"} Sep 16 13:58:27.584365 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.584255 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" event={"ID":"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43","Type":"ContainerStarted","Data":"5da2528cabcb88514736ba9267af83a7a67e2430f96a1717877ecc2878007456"} Sep 16 13:58:27.590984 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.590882 2580 scope.go:117] "RemoveContainer" containerID="5b89625eb7a716f4633e7fe2b520bc4f353c061246b8bf75750374098b88430c" Sep 16 13:58:27.591262 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:58:27.591243 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b89625eb7a716f4633e7fe2b520bc4f353c061246b8bf75750374098b88430c\": container with ID starting with 5b89625eb7a716f4633e7fe2b520bc4f353c061246b8bf75750374098b88430c not found: ID does not exist" containerID="5b89625eb7a716f4633e7fe2b520bc4f353c061246b8bf75750374098b88430c" Sep 16 13:58:27.591341 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.591275 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b89625eb7a716f4633e7fe2b520bc4f353c061246b8bf75750374098b88430c"} err="failed to get container status \"5b89625eb7a716f4633e7fe2b520bc4f353c061246b8bf75750374098b88430c\": rpc error: code = NotFound desc = could not find container \"5b89625eb7a716f4633e7fe2b520bc4f353c061246b8bf75750374098b88430c\": container with ID starting with 5b89625eb7a716f4633e7fe2b520bc4f353c061246b8bf75750374098b88430c not found: ID does not exist" Sep 16 13:58:27.591341 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.591300 2580 scope.go:117] "RemoveContainer" containerID="cf1104d9d676d623876d18c2b7281f2c550e83178c328099218c1063540f004e" Sep 16 13:58:27.615592 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.615543 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" podStartSLOduration=1.615515715 podStartE2EDuration="1.615515715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:58:27.615189261 +0000 UTC m=+1655.716002010" watchObservedRunningTime="2026-09-16 13:58:27.615515715 +0000 UTC m=+1655.716328430" Sep 16 13:58:27.636925 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.636892 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77"] Sep 16 13:58:27.640042 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.640017 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8c36a6/auth-test-model-8c36a6-kserve-c4d5bfdc9-45h77"] Sep 16 13:58:27.645724 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.645705 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:58:27.645811 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.645799 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12148179-76e7-4a24-bc06-932f8b764b43-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:58:27.645966 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.645952 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:58:27.646024 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.645975 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j5m9\" (UniqueName: \"kubernetes.io/projected/12148179-76e7-4a24-bc06-932f8b764b43-kube-api-access-9j5m9\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:58:27.646024 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.645992 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:58:27.646024 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.646010 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12148179-76e7-4a24-bc06-932f8b764b43-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:58:27.653080 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.653056 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx"] Sep 16 13:58:27.657824 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:27.657796 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-973610/auth-test-model-973610-kserve-6f57d4989f-sxskx"] Sep 16 13:58:28.355330 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:28.355291 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12148179-76e7-4a24-bc06-932f8b764b43" path="/var/lib/kubelet/pods/12148179-76e7-4a24-bc06-932f8b764b43/volumes" Sep 16 13:58:28.355691 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:28.355676 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39786424-c91c-427a-a41a-1e4758affc48" path="/var/lib/kubelet/pods/39786424-c91c-427a-a41a-1e4758affc48/volumes" Sep 16 13:58:29.960963 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:29.960930 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:58:37.054132 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:37.054079 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:37.054132 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:37.054146 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:37.067039 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:37.067009 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:37.646536 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:37.646505 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:58:49.976387 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:49.976350 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5"] Sep 16 13:58:49.977117 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:49.976695 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d852849-d15b-4de8-9b0c-3d4925a61534" containerName="istio-proxy" Sep 16 13:58:49.977117 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:49.976706 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d852849-d15b-4de8-9b0c-3d4925a61534" containerName="istio-proxy" Sep 16 13:58:49.977117 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:49.976722 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12148179-76e7-4a24-bc06-932f8b764b43" containerName="main" Sep 16 13:58:49.977117 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:49.976727 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="12148179-76e7-4a24-bc06-932f8b764b43" containerName="main" Sep 16 13:58:49.977117 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:49.976803 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="12148179-76e7-4a24-bc06-932f8b764b43" containerName="main" Sep 16 13:58:49.980711 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:49.980679 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:58:49.989581 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:49.989559 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8c36a6\"/\"sub-test-model-8c36a6-kserve-self-signed-certs\"" Sep 16 13:58:49.990630 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:49.990596 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8c36a6\"/\"kube-root-ca.crt\"" Sep 16 13:58:49.990784 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:49.990765 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8c36a6\"/\"default-dockercfg-9m5ns\"" Sep 16 13:58:49.990864 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:49.990824 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8c36a6\"/\"openshift-service-ca.crt\"" Sep 16 13:58:50.007817 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.007787 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5"] Sep 16 13:58:50.058159 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.058119 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-home\") pod \"sub-test-model-8c36a6-kserve-75db947d57-kw9p5\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:58:50.058340 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.058167 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-tmp-dir\") pod \"sub-test-model-8c36a6-kserve-75db947d57-kw9p5\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:58:50.058340 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.058199 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-model-cache\") pod \"sub-test-model-8c36a6-kserve-75db947d57-kw9p5\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:58:50.058340 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.058235 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d372b42f-a48c-45ca-a628-b589b130dab7-tls-certs\") pod \"sub-test-model-8c36a6-kserve-75db947d57-kw9p5\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:58:50.058340 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.058254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nxcn\" (UniqueName: \"kubernetes.io/projected/d372b42f-a48c-45ca-a628-b589b130dab7-kube-api-access-7nxcn\") pod \"sub-test-model-8c36a6-kserve-75db947d57-kw9p5\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:58:50.058483 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.058330 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-dshm\") pod \"sub-test-model-8c36a6-kserve-75db947d57-kw9p5\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:58:50.098719 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.098690 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:50.158893 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.158857 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-home\") pod \"sub-test-model-8c36a6-kserve-75db947d57-kw9p5\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:58:50.159094 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.158911 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-tmp-dir\") pod \"sub-test-model-8c36a6-kserve-75db947d57-kw9p5\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:58:50.159094 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.158937 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-model-cache\") pod \"sub-test-model-8c36a6-kserve-75db947d57-kw9p5\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:58:50.159094 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.158990 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d372b42f-a48c-45ca-a628-b589b130dab7-tls-certs\") pod \"sub-test-model-8c36a6-kserve-75db947d57-kw9p5\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:58:50.159094 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.159022 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nxcn\" (UniqueName: \"kubernetes.io/projected/d372b42f-a48c-45ca-a628-b589b130dab7-kube-api-access-7nxcn\") pod \"sub-test-model-8c36a6-kserve-75db947d57-kw9p5\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:58:50.159094 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.159055 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-dshm\") pod \"sub-test-model-8c36a6-kserve-75db947d57-kw9p5\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:58:50.159377 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.159324 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-home\") pod \"sub-test-model-8c36a6-kserve-75db947d57-kw9p5\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:58:50.159377 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.159360 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-tmp-dir\") pod \"sub-test-model-8c36a6-kserve-75db947d57-kw9p5\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:58:50.159475 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.159404 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-model-cache\") pod \"sub-test-model-8c36a6-kserve-75db947d57-kw9p5\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:58:50.161497 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.161477 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-dshm\") pod \"sub-test-model-8c36a6-kserve-75db947d57-kw9p5\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:58:50.161635 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.161597 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d372b42f-a48c-45ca-a628-b589b130dab7-tls-certs\") pod \"sub-test-model-8c36a6-kserve-75db947d57-kw9p5\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:58:50.171648 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.171624 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nxcn\" (UniqueName: \"kubernetes.io/projected/d372b42f-a48c-45ca-a628-b589b130dab7-kube-api-access-7nxcn\") pod \"sub-test-model-8c36a6-kserve-75db947d57-kw9p5\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:58:50.292462 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.292368 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:58:50.575223 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.575198 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5"] Sep 16 13:58:50.578065 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:58:50.578034 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd372b42f_a48c_45ca_a628_b589b130dab7.slice/crio-05ebb31cfe4b4348249803f26bbc4a8ae4f18001fc8c4ddadc617648090428d2 WatchSource:0}: Error finding container 05ebb31cfe4b4348249803f26bbc4a8ae4f18001fc8c4ddadc617648090428d2: Status 404 returned error can't find the container with id 05ebb31cfe4b4348249803f26bbc4a8ae4f18001fc8c4ddadc617648090428d2 Sep 16 13:58:50.701678 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.701637 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" event={"ID":"d372b42f-a48c-45ca-a628-b589b130dab7","Type":"ContainerStarted","Data":"3c3637ba3dccd53233c7b4f95ba3c286e276a3f89538bc78ce787a53088c47af"} Sep 16 13:58:50.701883 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.701691 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" event={"ID":"d372b42f-a48c-45ca-a628-b589b130dab7","Type":"ContainerStarted","Data":"05ebb31cfe4b4348249803f26bbc4a8ae4f18001fc8c4ddadc617648090428d2"} Sep 16 13:58:50.734063 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:50.734011 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" podStartSLOduration=1.733994577 podStartE2EDuration="1.733994577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:58:50.731129355 +0000 UTC m=+1678.831942168" watchObservedRunningTime="2026-09-16 13:58:50.733994577 +0000 UTC m=+1678.834807343" Sep 16 13:58:53.873927 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:58:53.873889 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:59:00.292897 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:00.292857 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:59:00.293353 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:00.293149 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:59:00.305469 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:00.305437 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:59:00.761202 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:00.761173 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:59:10.134670 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:10.134630 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:59:10.466293 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:10.466252 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:10.565686 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:10.565577 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:23.814505 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:23.814423 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:59:29.988994 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:29.988908 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:33.018076 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:33.018043 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9"] Sep 16 13:59:33.041529 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:33.041500 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:33.870980 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:33.870945 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5"] Sep 16 13:59:33.911311 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:33.911283 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:35.119402 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:35.119351 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:59:46.908284 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:46.908244 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567"] Sep 16 13:59:46.912131 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:46.912106 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:46.916474 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:46.916452 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-973610\"/\"rate-test-model-973610-kserve-self-signed-certs\"" Sep 16 13:59:46.929928 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:46.929902 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567"] Sep 16 13:59:46.951367 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:46.951343 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:46.986282 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:46.986254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8c24fa-e1f2-4345-8c83-9920c31afc02-tls-certs\") pod \"rate-test-model-973610-kserve-6475f98bf9-pp567\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:46.986420 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:46.986299 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m8h2\" (UniqueName: \"kubernetes.io/projected/cb8c24fa-e1f2-4345-8c83-9920c31afc02-kube-api-access-9m8h2\") pod \"rate-test-model-973610-kserve-6475f98bf9-pp567\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:46.986420 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:46.986323 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-model-cache\") pod \"rate-test-model-973610-kserve-6475f98bf9-pp567\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:46.986420 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:46.986371 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-tmp-dir\") pod \"rate-test-model-973610-kserve-6475f98bf9-pp567\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:46.986527 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:46.986420 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-dshm\") pod \"rate-test-model-973610-kserve-6475f98bf9-pp567\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:46.986527 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:46.986455 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-home\") pod \"rate-test-model-973610-kserve-6475f98bf9-pp567\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:47.086988 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:47.086935 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8c24fa-e1f2-4345-8c83-9920c31afc02-tls-certs\") pod \"rate-test-model-973610-kserve-6475f98bf9-pp567\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:47.087204 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:47.087005 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m8h2\" (UniqueName: \"kubernetes.io/projected/cb8c24fa-e1f2-4345-8c83-9920c31afc02-kube-api-access-9m8h2\") pod \"rate-test-model-973610-kserve-6475f98bf9-pp567\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:47.087204 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:47.087030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-model-cache\") pod \"rate-test-model-973610-kserve-6475f98bf9-pp567\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:47.087204 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:47.087048 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-tmp-dir\") pod \"rate-test-model-973610-kserve-6475f98bf9-pp567\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:47.087204 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:47.087065 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-dshm\") pod \"rate-test-model-973610-kserve-6475f98bf9-pp567\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:47.087204 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:47.087097 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-home\") pod \"rate-test-model-973610-kserve-6475f98bf9-pp567\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:47.087483 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:47.087462 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-tmp-dir\") pod \"rate-test-model-973610-kserve-6475f98bf9-pp567\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:47.087567 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:47.087535 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-model-cache\") pod \"rate-test-model-973610-kserve-6475f98bf9-pp567\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:47.087627 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:47.087573 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-home\") pod \"rate-test-model-973610-kserve-6475f98bf9-pp567\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:47.089536 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:47.089513 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-dshm\") pod \"rate-test-model-973610-kserve-6475f98bf9-pp567\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:47.089832 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:47.089814 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8c24fa-e1f2-4345-8c83-9920c31afc02-tls-certs\") pod \"rate-test-model-973610-kserve-6475f98bf9-pp567\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:47.096350 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:47.096329 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m8h2\" (UniqueName: \"kubernetes.io/projected/cb8c24fa-e1f2-4345-8c83-9920c31afc02-kube-api-access-9m8h2\") pod \"rate-test-model-973610-kserve-6475f98bf9-pp567\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:47.220874 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:47.220845 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:47.495198 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:47.495123 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567"] Sep 16 13:59:47.498579 ip-10-0-132-9 kubenswrapper[2580]: W0916 13:59:47.498551 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb8c24fa_e1f2_4345_8c83_9920c31afc02.slice/crio-813b39fc95a64d30fb1552f16099a6af06bc8c9de77cd6916684b52c3686f9e7 WatchSource:0}: Error finding container 813b39fc95a64d30fb1552f16099a6af06bc8c9de77cd6916684b52c3686f9e7: Status 404 returned error can't find the container with id 813b39fc95a64d30fb1552f16099a6af06bc8c9de77cd6916684b52c3686f9e7 Sep 16 13:59:47.973091 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:47.973055 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" event={"ID":"cb8c24fa-e1f2-4345-8c83-9920c31afc02","Type":"ContainerStarted","Data":"8c5578fa8d137f049ce0218447b065b85f7983d9b2be31a521a100b5af5def61"} Sep 16 13:59:47.973493 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:47.973097 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" event={"ID":"cb8c24fa-e1f2-4345-8c83-9920c31afc02","Type":"ContainerStarted","Data":"813b39fc95a64d30fb1552f16099a6af06bc8c9de77cd6916684b52c3686f9e7"} Sep 16 13:59:47.991913 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:47.991860 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" podStartSLOduration=1.9918419200000002 podStartE2EDuration="1.99184192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:59:47.991042166 +0000 UTC m=+1736.091854882" watchObservedRunningTime="2026-09-16 13:59:47.99184192 +0000 UTC m=+1736.092654679" Sep 16 13:59:48.037122 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.037085 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" podUID="26c3ca9b-22cc-430a-bcff-bfbf13bc4b43" containerName="main" containerID="cri-o://2b94774e47fb8b9ede4550244b4badd2a89b47663bb7e51dec6e5eecc1a86229" gracePeriod=45 Sep 16 13:59:48.277312 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.277288 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:59:48.297038 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.297010 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-dshm\") pod \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " Sep 16 13:59:48.297209 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.297066 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-tmp-dir\") pod \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " Sep 16 13:59:48.297209 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.297094 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-tls-certs\") pod \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " Sep 16 13:59:48.297209 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.297136 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smk8k\" (UniqueName: \"kubernetes.io/projected/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-kube-api-access-smk8k\") pod \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " Sep 16 13:59:48.297209 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.297158 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-model-cache\") pod \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " Sep 16 13:59:48.298435 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.297226 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-home\") pod \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\" (UID: \"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43\") " Sep 16 13:59:48.298435 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.297436 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "26c3ca9b-22cc-430a-bcff-bfbf13bc4b43" (UID: "26c3ca9b-22cc-430a-bcff-bfbf13bc4b43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:59:48.298435 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.298016 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:59:48.298435 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.298333 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-home" (OuterVolumeSpecName: "home") pod "26c3ca9b-22cc-430a-bcff-bfbf13bc4b43" (UID: "26c3ca9b-22cc-430a-bcff-bfbf13bc4b43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:59:48.298435 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.298433 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-model-cache" (OuterVolumeSpecName: "model-cache") pod "26c3ca9b-22cc-430a-bcff-bfbf13bc4b43" (UID: "26c3ca9b-22cc-430a-bcff-bfbf13bc4b43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:59:48.300008 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.299962 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-dshm" (OuterVolumeSpecName: "dshm") pod "26c3ca9b-22cc-430a-bcff-bfbf13bc4b43" (UID: "26c3ca9b-22cc-430a-bcff-bfbf13bc4b43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:59:48.300008 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.299970 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-kube-api-access-smk8k" (OuterVolumeSpecName: "kube-api-access-smk8k") pod "26c3ca9b-22cc-430a-bcff-bfbf13bc4b43" (UID: "26c3ca9b-22cc-430a-bcff-bfbf13bc4b43"). InnerVolumeSpecName "kube-api-access-smk8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:59:48.300434 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.300408 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26c3ca9b-22cc-430a-bcff-bfbf13bc4b43" (UID: "26c3ca9b-22cc-430a-bcff-bfbf13bc4b43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:59:48.399539 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.399498 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:59:48.399770 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.399640 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:59:48.399834 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.399771 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:59:48.399834 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.399794 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smk8k\" (UniqueName: \"kubernetes.io/projected/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-kube-api-access-smk8k\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:59:48.399834 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.399805 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:59:48.890746 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.890708 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" podUID="d372b42f-a48c-45ca-a628-b589b130dab7" containerName="main" containerID="cri-o://3c3637ba3dccd53233c7b4f95ba3c286e276a3f89538bc78ce787a53088c47af" gracePeriod=45 Sep 16 13:59:48.980852 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.980822 2580 generic.go:356] "Generic (PLEG): container finished" podID="26c3ca9b-22cc-430a-bcff-bfbf13bc4b43" containerID="2b94774e47fb8b9ede4550244b4badd2a89b47663bb7e51dec6e5eecc1a86229" exitCode=0 Sep 16 13:59:48.981237 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.980867 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" Sep 16 13:59:48.981237 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.980914 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" event={"ID":"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43","Type":"ContainerDied","Data":"2b94774e47fb8b9ede4550244b4badd2a89b47663bb7e51dec6e5eecc1a86229"} Sep 16 13:59:48.981237 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.980954 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9" event={"ID":"26c3ca9b-22cc-430a-bcff-bfbf13bc4b43","Type":"ContainerDied","Data":"5da2528cabcb88514736ba9267af83a7a67e2430f96a1717877ecc2878007456"} Sep 16 13:59:48.981237 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.980979 2580 scope.go:117] "RemoveContainer" containerID="2b94774e47fb8b9ede4550244b4badd2a89b47663bb7e51dec6e5eecc1a86229" Sep 16 13:59:48.997471 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:48.997445 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9"] Sep 16 13:59:49.005360 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.002193 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-973610/sub-test-model-973610-kserve-67f448679f-wmff9"] Sep 16 13:59:49.005360 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.002691 2580 scope.go:117] "RemoveContainer" containerID="2b94774e47fb8b9ede4550244b4badd2a89b47663bb7e51dec6e5eecc1a86229" Sep 16 13:59:49.005360 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:59:49.003168 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b94774e47fb8b9ede4550244b4badd2a89b47663bb7e51dec6e5eecc1a86229\": container with ID starting with 2b94774e47fb8b9ede4550244b4badd2a89b47663bb7e51dec6e5eecc1a86229 not found: ID does not exist" containerID="2b94774e47fb8b9ede4550244b4badd2a89b47663bb7e51dec6e5eecc1a86229" Sep 16 13:59:49.005360 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.003212 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b94774e47fb8b9ede4550244b4badd2a89b47663bb7e51dec6e5eecc1a86229"} err="failed to get container status \"2b94774e47fb8b9ede4550244b4badd2a89b47663bb7e51dec6e5eecc1a86229\": rpc error: code = NotFound desc = could not find container \"2b94774e47fb8b9ede4550244b4badd2a89b47663bb7e51dec6e5eecc1a86229\": container with ID starting with 2b94774e47fb8b9ede4550244b4badd2a89b47663bb7e51dec6e5eecc1a86229 not found: ID does not exist" Sep 16 13:59:49.152988 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.152916 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:59:49.211320 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.211283 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-home\") pod \"d372b42f-a48c-45ca-a628-b589b130dab7\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " Sep 16 13:59:49.211320 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.211320 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-dshm\") pod \"d372b42f-a48c-45ca-a628-b589b130dab7\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " Sep 16 13:59:49.211592 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.211367 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-model-cache\") pod \"d372b42f-a48c-45ca-a628-b589b130dab7\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " Sep 16 13:59:49.211592 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.211408 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d372b42f-a48c-45ca-a628-b589b130dab7-tls-certs\") pod \"d372b42f-a48c-45ca-a628-b589b130dab7\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " Sep 16 13:59:49.211592 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.211447 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-tmp-dir\") pod \"d372b42f-a48c-45ca-a628-b589b130dab7\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " Sep 16 13:59:49.211592 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.211488 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nxcn\" (UniqueName: \"kubernetes.io/projected/d372b42f-a48c-45ca-a628-b589b130dab7-kube-api-access-7nxcn\") pod \"d372b42f-a48c-45ca-a628-b589b130dab7\" (UID: \"d372b42f-a48c-45ca-a628-b589b130dab7\") " Sep 16 13:59:49.211832 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.211595 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-home" (OuterVolumeSpecName: "home") pod "d372b42f-a48c-45ca-a628-b589b130dab7" (UID: "d372b42f-a48c-45ca-a628-b589b130dab7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:59:49.211832 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.211736 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d372b42f-a48c-45ca-a628-b589b130dab7" (UID: "d372b42f-a48c-45ca-a628-b589b130dab7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:59:49.211832 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.211754 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-model-cache" (OuterVolumeSpecName: "model-cache") pod "d372b42f-a48c-45ca-a628-b589b130dab7" (UID: "d372b42f-a48c-45ca-a628-b589b130dab7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:59:49.211832 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.211762 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:59:49.213837 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.213802 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-dshm" (OuterVolumeSpecName: "dshm") pod "d372b42f-a48c-45ca-a628-b589b130dab7" (UID: "d372b42f-a48c-45ca-a628-b589b130dab7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:59:49.213837 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.213828 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d372b42f-a48c-45ca-a628-b589b130dab7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d372b42f-a48c-45ca-a628-b589b130dab7" (UID: "d372b42f-a48c-45ca-a628-b589b130dab7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:59:49.214003 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.213893 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d372b42f-a48c-45ca-a628-b589b130dab7-kube-api-access-7nxcn" (OuterVolumeSpecName: "kube-api-access-7nxcn") pod "d372b42f-a48c-45ca-a628-b589b130dab7" (UID: "d372b42f-a48c-45ca-a628-b589b130dab7"). InnerVolumeSpecName "kube-api-access-7nxcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:59:49.312723 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.312686 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d372b42f-a48c-45ca-a628-b589b130dab7-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:59:49.312723 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.312718 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:59:49.312723 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.312729 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nxcn\" (UniqueName: \"kubernetes.io/projected/d372b42f-a48c-45ca-a628-b589b130dab7-kube-api-access-7nxcn\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:59:49.312961 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.312738 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:59:49.312961 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.312747 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d372b42f-a48c-45ca-a628-b589b130dab7-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 13:59:49.987137 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.987105 2580 generic.go:356] "Generic (PLEG): container finished" podID="d372b42f-a48c-45ca-a628-b589b130dab7" containerID="3c3637ba3dccd53233c7b4f95ba3c286e276a3f89538bc78ce787a53088c47af" exitCode=0 Sep 16 13:59:49.987641 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.987147 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" event={"ID":"d372b42f-a48c-45ca-a628-b589b130dab7","Type":"ContainerDied","Data":"3c3637ba3dccd53233c7b4f95ba3c286e276a3f89538bc78ce787a53088c47af"} Sep 16 13:59:49.987641 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.987170 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" Sep 16 13:59:49.987641 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.987192 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5" event={"ID":"d372b42f-a48c-45ca-a628-b589b130dab7","Type":"ContainerDied","Data":"05ebb31cfe4b4348249803f26bbc4a8ae4f18001fc8c4ddadc617648090428d2"} Sep 16 13:59:49.987641 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.987215 2580 scope.go:117] "RemoveContainer" containerID="3c3637ba3dccd53233c7b4f95ba3c286e276a3f89538bc78ce787a53088c47af" Sep 16 13:59:49.996874 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.996840 2580 scope.go:117] "RemoveContainer" containerID="3c3637ba3dccd53233c7b4f95ba3c286e276a3f89538bc78ce787a53088c47af" Sep 16 13:59:49.997178 ip-10-0-132-9 kubenswrapper[2580]: E0916 13:59:49.997158 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c3637ba3dccd53233c7b4f95ba3c286e276a3f89538bc78ce787a53088c47af\": container with ID starting with 3c3637ba3dccd53233c7b4f95ba3c286e276a3f89538bc78ce787a53088c47af not found: ID does not exist" containerID="3c3637ba3dccd53233c7b4f95ba3c286e276a3f89538bc78ce787a53088c47af" Sep 16 13:59:49.997246 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:49.997187 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c3637ba3dccd53233c7b4f95ba3c286e276a3f89538bc78ce787a53088c47af"} err="failed to get container status \"3c3637ba3dccd53233c7b4f95ba3c286e276a3f89538bc78ce787a53088c47af\": rpc error: code = NotFound desc = could not find container \"3c3637ba3dccd53233c7b4f95ba3c286e276a3f89538bc78ce787a53088c47af\": container with ID starting with 3c3637ba3dccd53233c7b4f95ba3c286e276a3f89538bc78ce787a53088c47af not found: ID does not exist" Sep 16 13:59:50.010494 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:50.010461 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5"] Sep 16 13:59:50.016034 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:50.016006 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8c36a6/sub-test-model-8c36a6-kserve-75db947d57-kw9p5"] Sep 16 13:59:50.104866 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:50.104833 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 13:59:50.355146 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:50.355068 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26c3ca9b-22cc-430a-bcff-bfbf13bc4b43" path="/var/lib/kubelet/pods/26c3ca9b-22cc-430a-bcff-bfbf13bc4b43/volumes" Sep 16 13:59:50.355458 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:50.355445 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d372b42f-a48c-45ca-a628-b589b130dab7" path="/var/lib/kubelet/pods/d372b42f-a48c-45ca-a628-b589b130dab7/volumes" Sep 16 13:59:57.221023 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:57.220981 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:57.221481 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:57.221045 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:57.233493 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:57.233469 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 13:59:58.039652 ip-10-0-132-9 kubenswrapper[2580]: I0916 13:59:58.039600 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 14:00:00.174920 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:00.174878 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826120-ljkdk"] Sep 16 14:00:00.175413 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:00.175363 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26c3ca9b-22cc-430a-bcff-bfbf13bc4b43" containerName="main" Sep 16 14:00:00.175413 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:00.175382 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="26c3ca9b-22cc-430a-bcff-bfbf13bc4b43" containerName="main" Sep 16 14:00:00.175537 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:00.175430 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d372b42f-a48c-45ca-a628-b589b130dab7" containerName="main" Sep 16 14:00:00.175537 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:00.175439 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d372b42f-a48c-45ca-a628-b589b130dab7" containerName="main" Sep 16 14:00:00.175537 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:00.175528 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="26c3ca9b-22cc-430a-bcff-bfbf13bc4b43" containerName="main" Sep 16 14:00:00.176002 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:00.175541 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d372b42f-a48c-45ca-a628-b589b130dab7" containerName="main" Sep 16 14:00:00.178797 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:00.178775 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-ljkdk" Sep 16 14:00:00.182468 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:00.182445 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-thk5m\"" Sep 16 14:00:00.188486 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:00.188469 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-ljkdk" Sep 16 14:00:00.188632 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:00.188597 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c36a6-29826120-fnd8j"] Sep 16 14:00:00.191776 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:00.191758 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c36a6-29826120-fnd8j" Sep 16 14:00:00.209869 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:00.209793 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826120-ljkdk"] Sep 16 14:00:00.211936 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:00.211912 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c36a6-29826120-fnd8j"] Sep 16 14:00:00.220394 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:00.220366 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c36a6-29826120-fnd8j" Sep 16 14:00:00.475076 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:00.475045 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826120-ljkdk"] Sep 16 14:00:00.477383 ip-10-0-132-9 kubenswrapper[2580]: W0916 14:00:00.477357 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9c6684_c951_4ec8_bd65_aff71f653452.slice/crio-9fb01e2fe7480aba915434016322ec05fc706c7429b2b09c9c01c1a448025707 WatchSource:0}: Error finding container 9fb01e2fe7480aba915434016322ec05fc706c7429b2b09c9c01c1a448025707: Status 404 returned error can't find the container with id 9fb01e2fe7480aba915434016322ec05fc706c7429b2b09c9c01c1a448025707 Sep 16 14:00:00.504262 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:00.504213 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c36a6-29826120-fnd8j"] Sep 16 14:00:00.515769 ip-10-0-132-9 kubenswrapper[2580]: W0916 14:00:00.515744 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6562cc2c_077d_4cd4_b4af_18cac0a8020b.slice/crio-9a764e044c79a939de1550f8ce2c58ae1d0c885d68af646006b25100600df51d WatchSource:0}: Error finding container 9a764e044c79a939de1550f8ce2c58ae1d0c885d68af646006b25100600df51d: Status 404 returned error can't find the container with id 9a764e044c79a939de1550f8ce2c58ae1d0c885d68af646006b25100600df51d Sep 16 14:00:01.050483 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:01.050382 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-ljkdk" event={"ID":"0f9c6684-c951-4ec8-bd65-aff71f653452","Type":"ContainerStarted","Data":"ea4bedfd026afe725786105f16b1439ed8622bf3ce45e8e90eb8b379ab3a1dee"} Sep 16 14:00:01.050483 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:01.050474 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-ljkdk" event={"ID":"0f9c6684-c951-4ec8-bd65-aff71f653452","Type":"ContainerStarted","Data":"9fb01e2fe7480aba915434016322ec05fc706c7429b2b09c9c01c1a448025707"} Sep 16 14:00:01.052193 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:01.052165 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c36a6-29826120-fnd8j" event={"ID":"6562cc2c-077d-4cd4-b4af-18cac0a8020b","Type":"ContainerStarted","Data":"241bc7e24d968ce28ba37c1df7d8dfc1a4b19428ea553b39d6c8487962f5e0bf"} Sep 16 14:00:01.052299 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:01.052207 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c36a6-29826120-fnd8j" event={"ID":"6562cc2c-077d-4cd4-b4af-18cac0a8020b","Type":"ContainerStarted","Data":"9a764e044c79a939de1550f8ce2c58ae1d0c885d68af646006b25100600df51d"} Sep 16 14:00:01.072925 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:01.072871 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-ljkdk" podStartSLOduration=1.072856377 podStartE2EDuration="1.072856377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:01.07146084 +0000 UTC m=+1749.172273557" watchObservedRunningTime="2026-09-16 14:00:01.072856377 +0000 UTC m=+1749.173669089" Sep 16 14:00:01.093051 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:01.092995 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c36a6-29826120-fnd8j" podStartSLOduration=1.092977402 podStartE2EDuration="1.092977402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:01.092367885 +0000 UTC m=+1749.193180601" watchObservedRunningTime="2026-09-16 14:00:01.092977402 +0000 UTC m=+1749.193790118" Sep 16 14:00:02.057846 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:02.057814 2580 generic.go:356] "Generic (PLEG): container finished" podID="0f9c6684-c951-4ec8-bd65-aff71f653452" containerID="ea4bedfd026afe725786105f16b1439ed8622bf3ce45e8e90eb8b379ab3a1dee" exitCode=0 Sep 16 14:00:02.058282 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:02.057897 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-ljkdk" event={"ID":"0f9c6684-c951-4ec8-bd65-aff71f653452","Type":"ContainerDied","Data":"ea4bedfd026afe725786105f16b1439ed8622bf3ce45e8e90eb8b379ab3a1dee"} Sep 16 14:00:02.059433 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:02.059409 2580 generic.go:356] "Generic (PLEG): container finished" podID="6562cc2c-077d-4cd4-b4af-18cac0a8020b" containerID="241bc7e24d968ce28ba37c1df7d8dfc1a4b19428ea553b39d6c8487962f5e0bf" exitCode=0 Sep 16 14:00:02.059567 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:02.059487 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c36a6-29826120-fnd8j" event={"ID":"6562cc2c-077d-4cd4-b4af-18cac0a8020b","Type":"ContainerDied","Data":"241bc7e24d968ce28ba37c1df7d8dfc1a4b19428ea553b39d6c8487962f5e0bf"} Sep 16 14:00:03.212164 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:03.212141 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c36a6-29826120-fnd8j" Sep 16 14:00:03.215361 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:03.215342 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-ljkdk" Sep 16 14:00:04.071024 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:04.070995 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-ljkdk" Sep 16 14:00:04.071024 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:04.071013 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-ljkdk" event={"ID":"0f9c6684-c951-4ec8-bd65-aff71f653452","Type":"ContainerDied","Data":"9fb01e2fe7480aba915434016322ec05fc706c7429b2b09c9c01c1a448025707"} Sep 16 14:00:04.071277 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:04.071057 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fb01e2fe7480aba915434016322ec05fc706c7429b2b09c9c01c1a448025707" Sep 16 14:00:04.072279 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:04.072259 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c36a6-29826120-fnd8j" Sep 16 14:00:04.072401 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:04.072292 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c36a6-29826120-fnd8j" event={"ID":"6562cc2c-077d-4cd4-b4af-18cac0a8020b","Type":"ContainerDied","Data":"9a764e044c79a939de1550f8ce2c58ae1d0c885d68af646006b25100600df51d"} Sep 16 14:00:04.072401 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:04.072329 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a764e044c79a939de1550f8ce2c58ae1d0c885d68af646006b25100600df51d" Sep 16 14:00:16.880424 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:16.880391 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd"] Sep 16 14:00:16.880810 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:16.880753 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6562cc2c-077d-4cd4-b4af-18cac0a8020b" containerName="cleanup" Sep 16 14:00:16.880810 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:16.880766 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6562cc2c-077d-4cd4-b4af-18cac0a8020b" containerName="cleanup" Sep 16 14:00:16.880810 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:16.880780 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f9c6684-c951-4ec8-bd65-aff71f653452" containerName="cleanup" Sep 16 14:00:16.880810 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:16.880785 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f9c6684-c951-4ec8-bd65-aff71f653452" containerName="cleanup" Sep 16 14:00:16.880955 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:16.880844 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6562cc2c-077d-4cd4-b4af-18cac0a8020b" containerName="cleanup" Sep 16 14:00:16.880955 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:16.880857 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f9c6684-c951-4ec8-bd65-aff71f653452" containerName="cleanup" Sep 16 14:00:16.883760 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:16.883744 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:16.887629 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:16.887577 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8c36a6\"/\"kube-root-ca.crt\"" Sep 16 14:00:16.887629 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:16.887577 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8c36a6\"/\"default-dockercfg-9m5ns\"" Sep 16 14:00:16.887853 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:16.887632 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8c36a6\"/\"openshift-service-ca.crt\"" Sep 16 14:00:16.887853 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:16.887631 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8c36a6\"/\"rate-test-model-8c36a6-kserve-self-signed-certs\"" Sep 16 14:00:16.900962 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:16.900934 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd"] Sep 16 14:00:16.922413 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:16.922379 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:16.974405 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:16.974366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bm5p\" (UniqueName: \"kubernetes.io/projected/6f2618f9-5033-4f63-a096-52ddc22f583f-kube-api-access-8bm5p\") pod \"rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:16.974405 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:16.974409 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-home\") pod \"rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:16.974649 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:16.974429 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-dshm\") pod \"rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:16.974649 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:16.974476 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-tmp-dir\") pod \"rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:16.974649 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:16.974531 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2618f9-5033-4f63-a096-52ddc22f583f-tls-certs\") pod \"rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:16.974649 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:16.974568 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-model-cache\") pod \"rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:17.075460 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:17.075418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bm5p\" (UniqueName: \"kubernetes.io/projected/6f2618f9-5033-4f63-a096-52ddc22f583f-kube-api-access-8bm5p\") pod \"rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:17.075681 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:17.075475 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-home\") pod \"rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:17.075681 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:17.075500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-dshm\") pod \"rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:17.075681 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:17.075534 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-tmp-dir\") pod \"rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:17.075681 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:17.075583 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2618f9-5033-4f63-a096-52ddc22f583f-tls-certs\") pod \"rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:17.075681 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:17.075637 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-model-cache\") pod \"rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:17.076021 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:17.075996 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-home\") pod \"rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:17.076090 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:17.076040 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-tmp-dir\") pod \"rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:17.076090 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:17.076062 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-model-cache\") pod \"rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:17.078057 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:17.078038 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-dshm\") pod \"rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:17.078157 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:17.078137 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2618f9-5033-4f63-a096-52ddc22f583f-tls-certs\") pod \"rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:17.085462 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:17.085437 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bm5p\" (UniqueName: \"kubernetes.io/projected/6f2618f9-5033-4f63-a096-52ddc22f583f-kube-api-access-8bm5p\") pod \"rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:17.192992 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:17.192900 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:17.466967 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:17.466924 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd"] Sep 16 14:00:17.469200 ip-10-0-132-9 kubenswrapper[2580]: W0916 14:00:17.469159 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f2618f9_5033_4f63_a096_52ddc22f583f.slice/crio-5b317a67fb345cff7125af8626fa246a285bda3dcd547c92cff00c9248fa8aaa WatchSource:0}: Error finding container 5b317a67fb345cff7125af8626fa246a285bda3dcd547c92cff00c9248fa8aaa: Status 404 returned error can't find the container with id 5b317a67fb345cff7125af8626fa246a285bda3dcd547c92cff00c9248fa8aaa Sep 16 14:00:18.143867 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:18.143826 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" event={"ID":"6f2618f9-5033-4f63-a096-52ddc22f583f","Type":"ContainerStarted","Data":"072e2aa1e1528191f0ff6440d39ab4e9bab2fa52f19a6830f800354ee9c0335b"} Sep 16 14:00:18.143867 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:18.143873 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" event={"ID":"6f2618f9-5033-4f63-a096-52ddc22f583f","Type":"ContainerStarted","Data":"5b317a67fb345cff7125af8626fa246a285bda3dcd547c92cff00c9248fa8aaa"} Sep 16 14:00:18.169854 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:18.169801 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" podStartSLOduration=2.169787539 podStartE2EDuration="2.169787539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:18.168256525 +0000 UTC m=+1766.269069240" watchObservedRunningTime="2026-09-16 14:00:18.169787539 +0000 UTC m=+1766.270600255" Sep 16 14:00:20.503959 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:20.503918 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:00:27.193341 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:27.193303 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:27.193766 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:27.193357 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:27.205972 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:27.205938 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:28.206291 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:28.206264 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:00:37.097018 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:37.096974 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:00:46.515546 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:46.515460 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:00:53.273082 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:53.273046 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567"] Sep 16 14:00:53.296378 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:53.296348 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:54.279437 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:54.279399 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd"] Sep 16 14:00:54.304482 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:54.304456 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:54.317868 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:54.317833 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:00:54.778180 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:54.778138 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:55.195052 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:55.194960 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:55.347469 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:55.347433 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:56.991354 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:56.991321 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:58.073769 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:58.073735 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:58.665332 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:58.665297 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:59.362079 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:59.362050 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:59.444650 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:59.444595 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:59.673070 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:00:59.672978 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:00.527680 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:00.527650 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:00.608080 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:00.608049 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:00.780249 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:00.780171 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:01.225198 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.225167 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs"] Sep 16 14:01:01.228556 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.228537 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs" Sep 16 14:01:01.232347 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.232323 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b59887-dockercfg-2xvq2\"" Sep 16 14:01:01.232449 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.232346 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b59887\"" Sep 16 14:01:01.240685 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.240665 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs"] Sep 16 14:01:01.260040 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.260012 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d"] Sep 16 14:01:01.263255 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.263240 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d" Sep 16 14:01:01.277645 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.277597 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88f9b5c1-e57d-4eae-b095-b794482d80f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs\" (UID: \"88f9b5c1-e57d-4eae-b095-b794482d80f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs" Sep 16 14:01:01.277755 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.277738 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcqml\" (UniqueName: \"kubernetes.io/projected/88f9b5c1-e57d-4eae-b095-b794482d80f8-kube-api-access-zcqml\") pod \"payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs\" (UID: \"88f9b5c1-e57d-4eae-b095-b794482d80f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs" Sep 16 14:01:01.280282 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.280258 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d"] Sep 16 14:01:01.379028 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.378995 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b54wv\" (UniqueName: \"kubernetes.io/projected/b412d46f-77d5-49ee-bee0-d863a2a933ba-kube-api-access-b54wv\") pod \"payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d\" (UID: \"b412d46f-77d5-49ee-bee0-d863a2a933ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d" Sep 16 14:01:01.379028 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.379039 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcqml\" (UniqueName: \"kubernetes.io/projected/88f9b5c1-e57d-4eae-b095-b794482d80f8-kube-api-access-zcqml\") pod \"payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs\" (UID: \"88f9b5c1-e57d-4eae-b095-b794482d80f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs" Sep 16 14:01:01.379258 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.379099 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b412d46f-77d5-49ee-bee0-d863a2a933ba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d\" (UID: \"b412d46f-77d5-49ee-bee0-d863a2a933ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d" Sep 16 14:01:01.379258 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.379187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88f9b5c1-e57d-4eae-b095-b794482d80f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs\" (UID: \"88f9b5c1-e57d-4eae-b095-b794482d80f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs" Sep 16 14:01:01.379861 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.379844 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88f9b5c1-e57d-4eae-b095-b794482d80f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs\" (UID: \"88f9b5c1-e57d-4eae-b095-b794482d80f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs" Sep 16 14:01:01.391180 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.391150 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcqml\" (UniqueName: \"kubernetes.io/projected/88f9b5c1-e57d-4eae-b095-b794482d80f8-kube-api-access-zcqml\") pod \"payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs\" (UID: \"88f9b5c1-e57d-4eae-b095-b794482d80f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs" Sep 16 14:01:01.480048 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.479966 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b54wv\" (UniqueName: \"kubernetes.io/projected/b412d46f-77d5-49ee-bee0-d863a2a933ba-kube-api-access-b54wv\") pod \"payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d\" (UID: \"b412d46f-77d5-49ee-bee0-d863a2a933ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d" Sep 16 14:01:01.480048 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.480017 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b412d46f-77d5-49ee-bee0-d863a2a933ba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d\" (UID: \"b412d46f-77d5-49ee-bee0-d863a2a933ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d" Sep 16 14:01:01.480671 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.480652 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b412d46f-77d5-49ee-bee0-d863a2a933ba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d\" (UID: \"b412d46f-77d5-49ee-bee0-d863a2a933ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d" Sep 16 14:01:01.488550 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.488521 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b54wv\" (UniqueName: \"kubernetes.io/projected/b412d46f-77d5-49ee-bee0-d863a2a933ba-kube-api-access-b54wv\") pod \"payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d\" (UID: \"b412d46f-77d5-49ee-bee0-d863a2a933ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d" Sep 16 14:01:01.539000 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.538971 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs" Sep 16 14:01:01.573311 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.573284 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d" Sep 16 14:01:01.821228 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.821200 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs"] Sep 16 14:01:01.823158 ip-10-0-132-9 kubenswrapper[2580]: W0916 14:01:01.823133 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f9b5c1_e57d_4eae_b095_b794482d80f8.slice/crio-cec6c44ff66f7045ca2174fa93e3f4e9377a15bd516be9910460fabfc041ee55 WatchSource:0}: Error finding container cec6c44ff66f7045ca2174fa93e3f4e9377a15bd516be9910460fabfc041ee55: Status 404 returned error can't find the container with id cec6c44ff66f7045ca2174fa93e3f4e9377a15bd516be9910460fabfc041ee55 Sep 16 14:01:01.863311 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:01.863281 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d"] Sep 16 14:01:01.866911 ip-10-0-132-9 kubenswrapper[2580]: W0916 14:01:01.866887 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb412d46f_77d5_49ee_bee0_d863a2a933ba.slice/crio-3321251b5903bdea8a4cbf0fea0d4c5861ba062449923fe8f063165fe7e7119c WatchSource:0}: Error finding container 3321251b5903bdea8a4cbf0fea0d4c5861ba062449923fe8f063165fe7e7119c: Status 404 returned error can't find the container with id 3321251b5903bdea8a4cbf0fea0d4c5861ba062449923fe8f063165fe7e7119c Sep 16 14:01:02.362477 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:02.362444 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs" event={"ID":"88f9b5c1-e57d-4eae-b095-b794482d80f8","Type":"ContainerStarted","Data":"d37b58e986275d4eb18fe2779fb5ccdd8766bb99fe832650a2a11202cc103728"} Sep 16 14:01:02.362725 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:02.362503 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs" event={"ID":"88f9b5c1-e57d-4eae-b095-b794482d80f8","Type":"ContainerStarted","Data":"cec6c44ff66f7045ca2174fa93e3f4e9377a15bd516be9910460fabfc041ee55"} Sep 16 14:01:02.362725 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:02.362534 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs" Sep 16 14:01:02.364449 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:02.364419 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d" event={"ID":"b412d46f-77d5-49ee-bee0-d863a2a933ba","Type":"ContainerStarted","Data":"b6da5a6aa85e9994c1b483b5e37dd44af23f4ab5ce8a1356b17e9fc3d99e69d2"} Sep 16 14:01:02.364576 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:02.364479 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d" Sep 16 14:01:02.364576 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:02.364503 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d" event={"ID":"b412d46f-77d5-49ee-bee0-d863a2a933ba","Type":"ContainerStarted","Data":"3321251b5903bdea8a4cbf0fea0d4c5861ba062449923fe8f063165fe7e7119c"} Sep 16 14:01:02.382196 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:02.382142 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs" podStartSLOduration=1.382126693 podStartE2EDuration="1.382126693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:01:02.381291984 +0000 UTC m=+1810.482104701" watchObservedRunningTime="2026-09-16 14:01:02.382126693 +0000 UTC m=+1810.482939408" Sep 16 14:01:02.400683 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:02.400629 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d" podStartSLOduration=1.400594658 podStartE2EDuration="1.400594658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:01:02.399546901 +0000 UTC m=+1810.500359620" watchObservedRunningTime="2026-09-16 14:01:02.400594658 +0000 UTC m=+1810.501407376" Sep 16 14:01:04.363239 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:04.363202 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:05.410123 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:05.410087 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:01:08.292013 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:08.291969 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" podUID="cb8c24fa-e1f2-4345-8c83-9920c31afc02" containerName="main" containerID="cri-o://8c5578fa8d137f049ce0218447b065b85f7983d9b2be31a521a100b5af5def61" gracePeriod=45 Sep 16 14:01:08.545123 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:08.545062 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 14:01:08.645791 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:08.645753 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8c24fa-e1f2-4345-8c83-9920c31afc02-tls-certs\") pod \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " Sep 16 14:01:08.645791 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:08.645795 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9m8h2\" (UniqueName: \"kubernetes.io/projected/cb8c24fa-e1f2-4345-8c83-9920c31afc02-kube-api-access-9m8h2\") pod \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " Sep 16 14:01:08.646040 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:08.645830 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-model-cache\") pod \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " Sep 16 14:01:08.646040 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:08.645864 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-dshm\") pod \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " Sep 16 14:01:08.646040 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:08.645926 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-tmp-dir\") pod \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " Sep 16 14:01:08.646040 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:08.645985 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-home\") pod \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\" (UID: \"cb8c24fa-e1f2-4345-8c83-9920c31afc02\") " Sep 16 14:01:08.646268 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:08.646151 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-model-cache" (OuterVolumeSpecName: "model-cache") pod "cb8c24fa-e1f2-4345-8c83-9920c31afc02" (UID: "cb8c24fa-e1f2-4345-8c83-9920c31afc02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:01:08.646268 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:08.646249 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cb8c24fa-e1f2-4345-8c83-9920c31afc02" (UID: "cb8c24fa-e1f2-4345-8c83-9920c31afc02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:01:08.646353 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:08.646318 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:01:08.646353 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:08.646320 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-home" (OuterVolumeSpecName: "home") pod "cb8c24fa-e1f2-4345-8c83-9920c31afc02" (UID: "cb8c24fa-e1f2-4345-8c83-9920c31afc02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:01:08.648137 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:08.648110 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8c24fa-e1f2-4345-8c83-9920c31afc02-kube-api-access-9m8h2" (OuterVolumeSpecName: "kube-api-access-9m8h2") pod "cb8c24fa-e1f2-4345-8c83-9920c31afc02" (UID: "cb8c24fa-e1f2-4345-8c83-9920c31afc02"). InnerVolumeSpecName "kube-api-access-9m8h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:01:08.648513 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:08.648499 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb8c24fa-e1f2-4345-8c83-9920c31afc02-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cb8c24fa-e1f2-4345-8c83-9920c31afc02" (UID: "cb8c24fa-e1f2-4345-8c83-9920c31afc02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:01:08.648567 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:08.648523 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-dshm" (OuterVolumeSpecName: "dshm") pod "cb8c24fa-e1f2-4345-8c83-9920c31afc02" (UID: "cb8c24fa-e1f2-4345-8c83-9920c31afc02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:01:08.747567 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:08.747527 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:01:08.747567 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:08.747560 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8c24fa-e1f2-4345-8c83-9920c31afc02-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:01:08.747567 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:08.747571 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m8h2\" (UniqueName: \"kubernetes.io/projected/cb8c24fa-e1f2-4345-8c83-9920c31afc02-kube-api-access-9m8h2\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:01:08.747856 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:08.747581 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:01:08.747856 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:08.747597 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb8c24fa-e1f2-4345-8c83-9920c31afc02-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:01:09.299890 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.299846 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" podUID="6f2618f9-5033-4f63-a096-52ddc22f583f" containerName="main" containerID="cri-o://072e2aa1e1528191f0ff6440d39ab4e9bab2fa52f19a6830f800354ee9c0335b" gracePeriod=45 Sep 16 14:01:09.402165 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.402131 2580 generic.go:356] "Generic (PLEG): container finished" podID="cb8c24fa-e1f2-4345-8c83-9920c31afc02" containerID="8c5578fa8d137f049ce0218447b065b85f7983d9b2be31a521a100b5af5def61" exitCode=0 Sep 16 14:01:09.402357 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.402179 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" event={"ID":"cb8c24fa-e1f2-4345-8c83-9920c31afc02","Type":"ContainerDied","Data":"8c5578fa8d137f049ce0218447b065b85f7983d9b2be31a521a100b5af5def61"} Sep 16 14:01:09.402357 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.402193 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" Sep 16 14:01:09.402357 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.402208 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567" event={"ID":"cb8c24fa-e1f2-4345-8c83-9920c31afc02","Type":"ContainerDied","Data":"813b39fc95a64d30fb1552f16099a6af06bc8c9de77cd6916684b52c3686f9e7"} Sep 16 14:01:09.402357 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.402225 2580 scope.go:117] "RemoveContainer" containerID="8c5578fa8d137f049ce0218447b065b85f7983d9b2be31a521a100b5af5def61" Sep 16 14:01:09.412008 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.411964 2580 scope.go:117] "RemoveContainer" containerID="8c5578fa8d137f049ce0218447b065b85f7983d9b2be31a521a100b5af5def61" Sep 16 14:01:09.412379 ip-10-0-132-9 kubenswrapper[2580]: E0916 14:01:09.412356 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c5578fa8d137f049ce0218447b065b85f7983d9b2be31a521a100b5af5def61\": container with ID starting with 8c5578fa8d137f049ce0218447b065b85f7983d9b2be31a521a100b5af5def61 not found: ID does not exist" containerID="8c5578fa8d137f049ce0218447b065b85f7983d9b2be31a521a100b5af5def61" Sep 16 14:01:09.412458 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.412392 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c5578fa8d137f049ce0218447b065b85f7983d9b2be31a521a100b5af5def61"} err="failed to get container status \"8c5578fa8d137f049ce0218447b065b85f7983d9b2be31a521a100b5af5def61\": rpc error: code = NotFound desc = could not find container \"8c5578fa8d137f049ce0218447b065b85f7983d9b2be31a521a100b5af5def61\": container with ID starting with 8c5578fa8d137f049ce0218447b065b85f7983d9b2be31a521a100b5af5def61 not found: ID does not exist" Sep 16 14:01:09.424511 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.424482 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567"] Sep 16 14:01:09.427709 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.427684 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-973610/rate-test-model-973610-kserve-6475f98bf9-pp567"] Sep 16 14:01:09.548130 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.548105 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:01:09.655985 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.655864 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-tmp-dir\") pod \"6f2618f9-5033-4f63-a096-52ddc22f583f\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " Sep 16 14:01:09.655985 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.655951 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-model-cache\") pod \"6f2618f9-5033-4f63-a096-52ddc22f583f\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " Sep 16 14:01:09.656224 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.656013 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-home\") pod \"6f2618f9-5033-4f63-a096-52ddc22f583f\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " Sep 16 14:01:09.656224 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.656039 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bm5p\" (UniqueName: \"kubernetes.io/projected/6f2618f9-5033-4f63-a096-52ddc22f583f-kube-api-access-8bm5p\") pod \"6f2618f9-5033-4f63-a096-52ddc22f583f\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " Sep 16 14:01:09.656224 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.656087 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-dshm\") pod \"6f2618f9-5033-4f63-a096-52ddc22f583f\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " Sep 16 14:01:09.656224 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.656114 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2618f9-5033-4f63-a096-52ddc22f583f-tls-certs\") pod \"6f2618f9-5033-4f63-a096-52ddc22f583f\" (UID: \"6f2618f9-5033-4f63-a096-52ddc22f583f\") " Sep 16 14:01:09.656224 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.656147 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f2618f9-5033-4f63-a096-52ddc22f583f" (UID: "6f2618f9-5033-4f63-a096-52ddc22f583f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:01:09.656224 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.656195 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f2618f9-5033-4f63-a096-52ddc22f583f" (UID: "6f2618f9-5033-4f63-a096-52ddc22f583f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:01:09.656524 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.656295 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-home" (OuterVolumeSpecName: "home") pod "6f2618f9-5033-4f63-a096-52ddc22f583f" (UID: "6f2618f9-5033-4f63-a096-52ddc22f583f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:01:09.656524 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.656444 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:01:09.656524 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.656460 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:01:09.656524 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.656470 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:01:09.658407 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.658370 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f2618f9-5033-4f63-a096-52ddc22f583f-kube-api-access-8bm5p" (OuterVolumeSpecName: "kube-api-access-8bm5p") pod "6f2618f9-5033-4f63-a096-52ddc22f583f" (UID: "6f2618f9-5033-4f63-a096-52ddc22f583f"). InnerVolumeSpecName "kube-api-access-8bm5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:01:09.658517 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.658443 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f2618f9-5033-4f63-a096-52ddc22f583f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f2618f9-5033-4f63-a096-52ddc22f583f" (UID: "6f2618f9-5033-4f63-a096-52ddc22f583f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:01:09.658517 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.658443 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-dshm" (OuterVolumeSpecName: "dshm") pod "6f2618f9-5033-4f63-a096-52ddc22f583f" (UID: "6f2618f9-5033-4f63-a096-52ddc22f583f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:01:09.756934 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.756898 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bm5p\" (UniqueName: \"kubernetes.io/projected/6f2618f9-5033-4f63-a096-52ddc22f583f-kube-api-access-8bm5p\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:01:09.756934 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.756930 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2618f9-5033-4f63-a096-52ddc22f583f-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:01:09.756934 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:09.756939 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2618f9-5033-4f63-a096-52ddc22f583f-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:01:10.356590 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:10.356557 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb8c24fa-e1f2-4345-8c83-9920c31afc02" path="/var/lib/kubelet/pods/cb8c24fa-e1f2-4345-8c83-9920c31afc02/volumes" Sep 16 14:01:10.408810 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:10.408775 2580 generic.go:356] "Generic (PLEG): container finished" podID="6f2618f9-5033-4f63-a096-52ddc22f583f" containerID="072e2aa1e1528191f0ff6440d39ab4e9bab2fa52f19a6830f800354ee9c0335b" exitCode=0 Sep 16 14:01:10.409018 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:10.408819 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" Sep 16 14:01:10.409018 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:10.408859 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" event={"ID":"6f2618f9-5033-4f63-a096-52ddc22f583f","Type":"ContainerDied","Data":"072e2aa1e1528191f0ff6440d39ab4e9bab2fa52f19a6830f800354ee9c0335b"} Sep 16 14:01:10.409018 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:10.408905 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd" event={"ID":"6f2618f9-5033-4f63-a096-52ddc22f583f","Type":"ContainerDied","Data":"5b317a67fb345cff7125af8626fa246a285bda3dcd547c92cff00c9248fa8aaa"} Sep 16 14:01:10.409018 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:10.408922 2580 scope.go:117] "RemoveContainer" containerID="072e2aa1e1528191f0ff6440d39ab4e9bab2fa52f19a6830f800354ee9c0335b" Sep 16 14:01:10.417354 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:10.417333 2580 scope.go:117] "RemoveContainer" containerID="072e2aa1e1528191f0ff6440d39ab4e9bab2fa52f19a6830f800354ee9c0335b" Sep 16 14:01:10.417676 ip-10-0-132-9 kubenswrapper[2580]: E0916 14:01:10.417656 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"072e2aa1e1528191f0ff6440d39ab4e9bab2fa52f19a6830f800354ee9c0335b\": container with ID starting with 072e2aa1e1528191f0ff6440d39ab4e9bab2fa52f19a6830f800354ee9c0335b not found: ID does not exist" containerID="072e2aa1e1528191f0ff6440d39ab4e9bab2fa52f19a6830f800354ee9c0335b" Sep 16 14:01:10.417747 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:10.417685 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"072e2aa1e1528191f0ff6440d39ab4e9bab2fa52f19a6830f800354ee9c0335b"} err="failed to get container status \"072e2aa1e1528191f0ff6440d39ab4e9bab2fa52f19a6830f800354ee9c0335b\": rpc error: code = NotFound desc = could not find container \"072e2aa1e1528191f0ff6440d39ab4e9bab2fa52f19a6830f800354ee9c0335b\": container with ID starting with 072e2aa1e1528191f0ff6440d39ab4e9bab2fa52f19a6830f800354ee9c0335b not found: ID does not exist" Sep 16 14:01:10.435842 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:10.435812 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd"] Sep 16 14:01:10.441478 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:10.441453 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8c36a6/rate-test-model-8c36a6-kserve-78b9d59b67-lv6dd"] Sep 16 14:01:12.354692 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:12.354657 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f2618f9-5033-4f63-a096-52ddc22f583f" path="/var/lib/kubelet/pods/6f2618f9-5033-4f63-a096-52ddc22f583f/volumes" Sep 16 14:01:13.371558 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:13.371527 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d" Sep 16 14:01:13.371992 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:13.371881 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs" Sep 16 14:01:13.872807 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:13.872774 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:21.554906 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:21.554858 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:23.347918 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:23.347876 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:23.733194 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:23.733165 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:24.534888 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.534843 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:24.637431 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.637401 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:24.690948 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.690915 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845"] Sep 16 14:01:24.691299 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.691285 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f2618f9-5033-4f63-a096-52ddc22f583f" containerName="main" Sep 16 14:01:24.691378 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.691303 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f2618f9-5033-4f63-a096-52ddc22f583f" containerName="main" Sep 16 14:01:24.691378 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.691318 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb8c24fa-e1f2-4345-8c83-9920c31afc02" containerName="main" Sep 16 14:01:24.691378 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.691324 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb8c24fa-e1f2-4345-8c83-9920c31afc02" containerName="main" Sep 16 14:01:24.691544 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.691381 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb8c24fa-e1f2-4345-8c83-9920c31afc02" containerName="main" Sep 16 14:01:24.691544 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.691394 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f2618f9-5033-4f63-a096-52ddc22f583f" containerName="main" Sep 16 14:01:24.694447 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.694421 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.697275 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.697245 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ac4a3c-openshift-default-dockercfg-69mm5\"" Sep 16 14:01:24.711366 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.711338 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845"] Sep 16 14:01:24.737457 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.737417 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:24.798647 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.798535 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-workload-certs\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.798647 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.798592 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d1c586-e589-4690-a507-98f8f26c5c42-istiod-ca-cert\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.798855 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.798718 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2df5\" (UniqueName: \"kubernetes.io/projected/a8d1c586-e589-4690-a507-98f8f26c5c42-kube-api-access-f2df5\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.798855 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.798764 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-envoy\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.798855 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.798849 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-podinfo\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.798974 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.798869 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-token\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.798974 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.798960 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-data\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.799050 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.798994 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-credential-socket\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.799090 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.799048 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-workload-socket\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.900397 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.900362 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-workload-socket\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.900598 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.900411 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-workload-certs\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.900598 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.900433 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d1c586-e589-4690-a507-98f8f26c5c42-istiod-ca-cert\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.900598 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.900457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2df5\" (UniqueName: \"kubernetes.io/projected/a8d1c586-e589-4690-a507-98f8f26c5c42-kube-api-access-f2df5\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.900598 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.900577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-envoy\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.900873 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.900675 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-podinfo\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.900873 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.900708 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-token\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.900873 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.900794 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-data\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.900873 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.900816 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-workload-socket\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.900873 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.900825 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-credential-socket\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.900873 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.900859 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-workload-certs\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.901197 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.901078 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-credential-socket\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.901197 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.901123 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-data\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.901304 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.901252 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d1c586-e589-4690-a507-98f8f26c5c42-istiod-ca-cert\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.902940 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.902922 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-envoy\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.903301 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.903284 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-podinfo\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.910496 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.910448 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-token\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:24.910637 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:24.910617 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2df5\" (UniqueName: \"kubernetes.io/projected/a8d1c586-e589-4690-a507-98f8f26c5c42-kube-api-access-f2df5\") pod \"e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:25.006256 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:25.006225 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:25.166850 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:25.166816 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:25.267051 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:25.267019 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:25.288567 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:25.288542 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845"] Sep 16 14:01:25.290248 ip-10-0-132-9 kubenswrapper[2580]: W0916 14:01:25.290209 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d1c586_e589_4690_a507_98f8f26c5c42.slice/crio-d104dcf703dfc48696d5a0c64509bdb03d86898edcc8d6b20ead9be20e9a509b WatchSource:0}: Error finding container d104dcf703dfc48696d5a0c64509bdb03d86898edcc8d6b20ead9be20e9a509b: Status 404 returned error can't find the container with id d104dcf703dfc48696d5a0c64509bdb03d86898edcc8d6b20ead9be20e9a509b Sep 16 14:01:25.292460 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:25.292426 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:01:25.292550 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:25.292493 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:01:25.292550 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:25.292520 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:01:25.489517 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:25.489485 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" event={"ID":"a8d1c586-e589-4690-a507-98f8f26c5c42","Type":"ContainerStarted","Data":"30211464e3d639197bd9e5ebed67552628061067a20f7dad87161392b33732d8"} Sep 16 14:01:25.489694 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:25.489539 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" event={"ID":"a8d1c586-e589-4690-a507-98f8f26c5c42","Type":"ContainerStarted","Data":"d104dcf703dfc48696d5a0c64509bdb03d86898edcc8d6b20ead9be20e9a509b"} Sep 16 14:01:25.517106 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:25.517048 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" podStartSLOduration=1.517030952 podStartE2EDuration="1.517030952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:01:25.515892202 +0000 UTC m=+1833.616704970" watchObservedRunningTime="2026-09-16 14:01:25.517030952 +0000 UTC m=+1833.617843666" Sep 16 14:01:25.757446 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:25.757413 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:26.006755 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:26.006719 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:26.012164 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:26.012090 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:26.495031 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:26.495003 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:26.496131 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:26.496112 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:26.886143 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:26.886049 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:27.479343 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:27.479304 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:27.573865 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:27.573825 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:27.679573 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:27.679539 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:28.526886 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:28.526853 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:29.403675 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:29.403643 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:30.152981 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:30.152944 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:31.052549 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:31.052517 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:31.142527 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:31.142495 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:34.309513 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:34.309477 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:01:42.496064 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:42.496014 2580 patch_prober.go:28] interesting pod/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 16 14:01:42.496545 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:42.496084 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" podUID="a8d1c586-e589-4690-a507-98f8f26c5c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 16 14:01:44.607169 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:44.607132 2580 generic.go:356] "Generic (PLEG): container finished" podID="a8d1c586-e589-4690-a507-98f8f26c5c42" containerID="30211464e3d639197bd9e5ebed67552628061067a20f7dad87161392b33732d8" exitCode=0 Sep 16 14:01:44.607629 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:44.607204 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" event={"ID":"a8d1c586-e589-4690-a507-98f8f26c5c42","Type":"ContainerDied","Data":"30211464e3d639197bd9e5ebed67552628061067a20f7dad87161392b33732d8"} Sep 16 14:01:44.607629 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:44.607586 2580 scope.go:117] "RemoveContainer" containerID="30211464e3d639197bd9e5ebed67552628061067a20f7dad87161392b33732d8" Sep 16 14:01:44.608659 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:44.608627 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:01:44.608773 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:44.608712 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:01:44.608773 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:44.608760 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:01:45.614644 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:45.614593 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" event={"ID":"a8d1c586-e589-4690-a507-98f8f26c5c42","Type":"ContainerStarted","Data":"60f66969db0e25fe603e91b1e8021810617938346d8bbbb263f408a12f8e1e3f"} Sep 16 14:01:46.007362 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:46.007321 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:46.007559 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:46.007524 2580 patch_prober.go:28] interesting pod/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Sep 16 14:01:46.007639 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:46.007582 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" podUID="a8d1c586-e589-4690-a507-98f8f26c5c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Sep 16 14:01:47.007469 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:47.007427 2580 patch_prober.go:28] interesting pod/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Sep 16 14:01:47.007923 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:47.007488 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" podUID="a8d1c586-e589-4690-a507-98f8f26c5c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Sep 16 14:01:48.007369 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:48.007329 2580 patch_prober.go:28] interesting pod/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Sep 16 14:01:48.007555 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:48.007401 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" podUID="a8d1c586-e589-4690-a507-98f8f26c5c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Sep 16 14:01:49.007267 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:49.007227 2580 patch_prober.go:28] interesting pod/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Sep 16 14:01:49.007455 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:49.007290 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" podUID="a8d1c586-e589-4690-a507-98f8f26c5c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Sep 16 14:01:49.034643 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:49.034597 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:50.006674 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:50.006630 2580 patch_prober.go:28] interesting pod/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Sep 16 14:01:50.006863 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:50.006722 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" podUID="a8d1c586-e589-4690-a507-98f8f26c5c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Sep 16 14:01:51.010320 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:51.010290 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:51.010796 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:51.010552 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:51.011291 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:51.011271 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:01:51.124451 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:51.124418 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:52.516922 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:52.516883 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:52.647461 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:52.647430 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:53.631339 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:53.631308 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:54.421377 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:54.421345 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:55.346168 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:55.346137 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:01:55.730065 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:01:55.730035 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:25.485577 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.485502 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45"] Sep 16 14:02:25.487901 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.487884 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:25.491821 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.491798 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b59887\"/\"default-dockercfg-xgc7f\"" Sep 16 14:02:25.491957 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.491798 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b59887\"/\"ipp-route-b59887-kserve-self-signed-certs\"" Sep 16 14:02:25.491957 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.491798 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b59887\"/\"openshift-service-ca.crt\"" Sep 16 14:02:25.491957 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.491805 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b59887\"/\"kube-root-ca.crt\"" Sep 16 14:02:25.500584 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.500562 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45"] Sep 16 14:02:25.519326 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.519284 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:25.574051 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.574022 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxwpf\" (UniqueName: \"kubernetes.io/projected/4e6d42b5-2015-4d39-8647-2fa231dcc643-kube-api-access-kxwpf\") pod \"ipp-route-b59887-kserve-67b98f8db7-w2j45\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:25.574200 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.574063 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-home\") pod \"ipp-route-b59887-kserve-67b98f8db7-w2j45\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:25.574200 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.574089 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-tmp-dir\") pod \"ipp-route-b59887-kserve-67b98f8db7-w2j45\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:25.574200 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.574157 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e6d42b5-2015-4d39-8647-2fa231dcc643-tls-certs\") pod \"ipp-route-b59887-kserve-67b98f8db7-w2j45\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:25.574200 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.574175 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-model-cache\") pod \"ipp-route-b59887-kserve-67b98f8db7-w2j45\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:25.574340 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.574216 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-dshm\") pod \"ipp-route-b59887-kserve-67b98f8db7-w2j45\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:25.674748 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.674718 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-dshm\") pod \"ipp-route-b59887-kserve-67b98f8db7-w2j45\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:25.674901 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.674764 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxwpf\" (UniqueName: \"kubernetes.io/projected/4e6d42b5-2015-4d39-8647-2fa231dcc643-kube-api-access-kxwpf\") pod \"ipp-route-b59887-kserve-67b98f8db7-w2j45\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:25.674965 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.674906 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-home\") pod \"ipp-route-b59887-kserve-67b98f8db7-w2j45\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:25.675002 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.674982 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-tmp-dir\") pod \"ipp-route-b59887-kserve-67b98f8db7-w2j45\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:25.675073 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.675054 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e6d42b5-2015-4d39-8647-2fa231dcc643-tls-certs\") pod \"ipp-route-b59887-kserve-67b98f8db7-w2j45\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:25.675112 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.675088 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-model-cache\") pod \"ipp-route-b59887-kserve-67b98f8db7-w2j45\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:25.675293 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.675262 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-home\") pod \"ipp-route-b59887-kserve-67b98f8db7-w2j45\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:25.675293 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.675280 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-tmp-dir\") pod \"ipp-route-b59887-kserve-67b98f8db7-w2j45\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:25.675486 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.675463 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-model-cache\") pod \"ipp-route-b59887-kserve-67b98f8db7-w2j45\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:25.677197 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.677176 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-dshm\") pod \"ipp-route-b59887-kserve-67b98f8db7-w2j45\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:25.677674 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.677656 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e6d42b5-2015-4d39-8647-2fa231dcc643-tls-certs\") pod \"ipp-route-b59887-kserve-67b98f8db7-w2j45\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:25.683597 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.683576 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxwpf\" (UniqueName: \"kubernetes.io/projected/4e6d42b5-2015-4d39-8647-2fa231dcc643-kube-api-access-kxwpf\") pod \"ipp-route-b59887-kserve-67b98f8db7-w2j45\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:25.797831 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:25.797746 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:26.070926 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:26.070841 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45"] Sep 16 14:02:26.074349 ip-10-0-132-9 kubenswrapper[2580]: W0916 14:02:26.074320 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e6d42b5_2015_4d39_8647_2fa231dcc643.slice/crio-00334c5b4487dd829ce22b965297ee040ccbb95ee56c24e7a90b7f7329712fb5 WatchSource:0}: Error finding container 00334c5b4487dd829ce22b965297ee040ccbb95ee56c24e7a90b7f7329712fb5: Status 404 returned error can't find the container with id 00334c5b4487dd829ce22b965297ee040ccbb95ee56c24e7a90b7f7329712fb5 Sep 16 14:02:26.821960 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:26.821923 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" event={"ID":"4e6d42b5-2015-4d39-8647-2fa231dcc643","Type":"ContainerStarted","Data":"688c6812cc8c78cf03a6fed305de9ba9bfc1bf5fc8385a86de930d6a872a893a"} Sep 16 14:02:26.822421 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:26.821970 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" event={"ID":"4e6d42b5-2015-4d39-8647-2fa231dcc643","Type":"ContainerStarted","Data":"00334c5b4487dd829ce22b965297ee040ccbb95ee56c24e7a90b7f7329712fb5"} Sep 16 14:02:26.848363 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:26.848315 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" podStartSLOduration=1.848299258 podStartE2EDuration="1.848299258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:02:26.847212882 +0000 UTC m=+1894.948025597" watchObservedRunningTime="2026-09-16 14:02:26.848299258 +0000 UTC m=+1894.949111972" Sep 16 14:02:29.313405 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:29.313369 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:02:35.798009 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:35.797970 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:35.798420 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:35.798023 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:35.810585 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:35.810559 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:35.878562 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:35.878528 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:02:42.715481 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:42.715448 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:02:48.920789 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:48.920759 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d_b412d46f-77d5-49ee-bee0-d863a2a933ba/payload-processing/0.log" Sep 16 14:02:49.450516 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:49.450482 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:49.741238 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:49.741197 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44"] Sep 16 14:02:49.743756 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:49.743733 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:49.746560 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:49.746541 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-bf580f-openshift-default-dockercfg-m8d46\"" Sep 16 14:02:49.759473 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:49.759449 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44"] Sep 16 14:02:49.763563 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:49.763532 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:49.902828 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:49.902789 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-podinfo\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:49.903033 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:49.902843 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-data\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:49.903033 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:49.902900 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-credential-socket\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:49.903033 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:49.902942 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjr9g\" (UniqueName: \"kubernetes.io/projected/ee2177e4-982f-488b-a1b9-25cc487b2e68-kube-api-access-sjr9g\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:49.903033 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:49.902968 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee2177e4-982f-488b-a1b9-25cc487b2e68-istiod-ca-cert\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:49.903033 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:49.902998 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-workload-socket\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:49.903235 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:49.903037 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-envoy\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:49.903235 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:49.903079 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-token\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:49.903235 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:49.903115 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-workload-certs\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:50.003939 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.003859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-envoy\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:50.003939 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.003903 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-token\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:50.003939 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.003928 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-workload-certs\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:50.004505 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.003955 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-podinfo\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:50.004505 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.003993 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-data\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:50.004505 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.004016 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-credential-socket\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:50.004505 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.004045 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjr9g\" (UniqueName: \"kubernetes.io/projected/ee2177e4-982f-488b-a1b9-25cc487b2e68-kube-api-access-sjr9g\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:50.004505 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.004082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee2177e4-982f-488b-a1b9-25cc487b2e68-istiod-ca-cert\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:50.004505 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.004141 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-workload-socket\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:50.004505 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.004414 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-credential-socket\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:50.004505 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.004458 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-workload-certs\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:50.004864 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.004627 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-workload-socket\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:50.004864 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.004648 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-data\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:50.005087 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.005069 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee2177e4-982f-488b-a1b9-25cc487b2e68-istiod-ca-cert\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:50.006486 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.006452 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-envoy\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:50.006719 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.006700 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-podinfo\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:50.010512 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.010476 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:50.012886 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.012864 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-token\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:50.012983 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.012874 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjr9g\" (UniqueName: \"kubernetes.io/projected/ee2177e4-982f-488b-a1b9-25cc487b2e68-kube-api-access-sjr9g\") pod \"e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:50.054823 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.054784 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:50.345392 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.345365 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44"] Sep 16 14:02:50.347127 ip-10-0-132-9 kubenswrapper[2580]: W0916 14:02:50.347099 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee2177e4_982f_488b_a1b9_25cc487b2e68.slice/crio-78f98b643b699dc0a6d7a01bc64a595c82dafc624b51fcd52eb77d2a8b9e870b WatchSource:0}: Error finding container 78f98b643b699dc0a6d7a01bc64a595c82dafc624b51fcd52eb77d2a8b9e870b: Status 404 returned error can't find the container with id 78f98b643b699dc0a6d7a01bc64a595c82dafc624b51fcd52eb77d2a8b9e870b Sep 16 14:02:50.349493 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.349459 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:02:50.349636 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.349532 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:02:50.349636 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.349559 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:02:50.953253 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.953214 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" event={"ID":"ee2177e4-982f-488b-a1b9-25cc487b2e68","Type":"ContainerStarted","Data":"c8d380af5738d3182450e0622c80d20dbe51b791eea3ad65202eb975c4d8e25a"} Sep 16 14:02:50.953461 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.953270 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" event={"ID":"ee2177e4-982f-488b-a1b9-25cc487b2e68","Type":"ContainerStarted","Data":"78f98b643b699dc0a6d7a01bc64a595c82dafc624b51fcd52eb77d2a8b9e870b"} Sep 16 14:02:50.976682 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:50.976632 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" podStartSLOduration=1.97659268 podStartE2EDuration="1.97659268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:02:50.975005511 +0000 UTC m=+1919.075818228" watchObservedRunningTime="2026-09-16 14:02:50.97659268 +0000 UTC m=+1919.077405395" Sep 16 14:02:51.055651 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:51.055596 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:51.060844 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:51.060814 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:51.958667 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:51.958634 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:51.959573 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:51.959543 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:02:52.229272 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:52.229235 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:52.793738 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:52.793711 2580 scope.go:117] "RemoveContainer" containerID="7b29b75e802ed23c5a59e4e976009f5fa7e020b2bebc43acb935a7fc97da5a4c" Sep 16 14:02:52.802682 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:52.802661 2580 scope.go:117] "RemoveContainer" containerID="17d7f8914706329984193e85adef87e6ba93008bf8cc8bd9a3429daa99890812" Sep 16 14:02:52.810712 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:52.810692 2580 scope.go:117] "RemoveContainer" containerID="0a95338b6266bcc7b6f6ac8817b8e9770fecc9041cd922175bfbb91f56a9a15b" Sep 16 14:02:52.818892 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:52.818872 2580 scope.go:117] "RemoveContainer" containerID="4efdfa847da52983bd7c1944fac6a3b9bcafbc14e44fff20865d83254a4fb490" Sep 16 14:02:52.942507 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:52.942473 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:53.038040 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:53.038008 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:54.080356 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:54.080318 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:54.181234 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:54.181199 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:55.046700 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:55.046666 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:55.734953 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:55.734918 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:56.636453 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:56.636418 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:56.693552 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:56.693512 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:56.794765 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:56.794718 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:57.127797 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:57.127763 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:02:58.262248 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:02:58.262215 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:03.039941 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:03.039908 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:03:14.064545 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:14.064510 2580 generic.go:356] "Generic (PLEG): container finished" podID="ee2177e4-982f-488b-a1b9-25cc487b2e68" containerID="c8d380af5738d3182450e0622c80d20dbe51b791eea3ad65202eb975c4d8e25a" exitCode=0 Sep 16 14:03:14.064978 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:14.064578 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" event={"ID":"ee2177e4-982f-488b-a1b9-25cc487b2e68","Type":"ContainerDied","Data":"c8d380af5738d3182450e0622c80d20dbe51b791eea3ad65202eb975c4d8e25a"} Sep 16 14:03:14.065031 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:14.064993 2580 scope.go:117] "RemoveContainer" containerID="c8d380af5738d3182450e0622c80d20dbe51b791eea3ad65202eb975c4d8e25a" Sep 16 14:03:14.066101 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:14.066068 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:03:14.066206 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:14.066158 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:03:14.066266 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:14.066206 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 14:03:15.075879 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:15.075829 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" event={"ID":"ee2177e4-982f-488b-a1b9-25cc487b2e68","Type":"ContainerStarted","Data":"9e674b54a71c9726c39964155027648cf17e16e35fe56f7f4b5ee8c7e18a92ef"} Sep 16 14:03:16.055327 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:16.055282 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:03:16.055523 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:16.055483 2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Sep 16 14:03:16.055578 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:16.055542 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" podUID="ee2177e4-982f-488b-a1b9-25cc487b2e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Sep 16 14:03:17.055399 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:17.055362 2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Sep 16 14:03:17.055834 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:17.055433 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" podUID="ee2177e4-982f-488b-a1b9-25cc487b2e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Sep 16 14:03:18.055944 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:18.055912 2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Sep 16 14:03:18.056330 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:18.055977 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" podUID="ee2177e4-982f-488b-a1b9-25cc487b2e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Sep 16 14:03:19.055792 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:19.055754 2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Sep 16 14:03:19.055977 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:19.055808 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" podUID="ee2177e4-982f-488b-a1b9-25cc487b2e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Sep 16 14:03:19.639643 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:19.639593 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:19.740314 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:19.740279 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:20.056013 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:20.055973 2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Sep 16 14:03:20.056402 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:20.056048 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" podUID="ee2177e4-982f-488b-a1b9-25cc487b2e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Sep 16 14:03:20.350312 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:20.350223 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:21.052183 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:21.052149 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:21.059190 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:21.059160 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:03:21.059642 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:21.059424 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:03:21.060069 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:21.060054 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:03:21.951572 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:21.951533 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:22.751977 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:22.751947 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:23.022344 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:23.022260 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:23.711989 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:23.711955 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:23.766737 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:23.766701 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bf580f-hk8p6"] Sep 16 14:03:23.770320 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:23.770301 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bf580f-hk8p6" Sep 16 14:03:23.775249 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:23.775227 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-thk5m\"" Sep 16 14:03:23.775398 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:23.775290 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 14:03:23.789623 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:23.789574 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bf580f-hk8p6"] Sep 16 14:03:23.870879 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:23.870843 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:23.930053 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:23.930023 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91df75b6-e934-4f46-840e-a4c9f75cba5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-bf580f-hk8p6\" (UID: \"91df75b6-e934-4f46-840e-a4c9f75cba5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bf580f-hk8p6" Sep 16 14:03:24.031184 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:24.031155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91df75b6-e934-4f46-840e-a4c9f75cba5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-bf580f-hk8p6\" (UID: \"91df75b6-e934-4f46-840e-a4c9f75cba5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bf580f-hk8p6" Sep 16 14:03:24.031862 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:24.031835 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91df75b6-e934-4f46-840e-a4c9f75cba5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-bf580f-hk8p6\" (UID: \"91df75b6-e934-4f46-840e-a4c9f75cba5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bf580f-hk8p6" Sep 16 14:03:24.080597 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:24.080557 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bf580f-hk8p6" Sep 16 14:03:24.358645 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:24.357203 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bf580f-hk8p6"] Sep 16 14:03:24.362939 ip-10-0-132-9 kubenswrapper[2580]: W0916 14:03:24.362901 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91df75b6_e934_4f46_840e_a4c9f75cba5b.slice/crio-89cc5bb5a20d4cb302cb9746674c0e5ddf8c961941010b0fa06b739bf7a0b6da WatchSource:0}: Error finding container 89cc5bb5a20d4cb302cb9746674c0e5ddf8c961941010b0fa06b739bf7a0b6da: Status 404 returned error can't find the container with id 89cc5bb5a20d4cb302cb9746674c0e5ddf8c961941010b0fa06b739bf7a0b6da Sep 16 14:03:24.668935 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:24.668896 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:25.137123 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:25.137082 2580 generic.go:356] "Generic (PLEG): container finished" podID="91df75b6-e934-4f46-840e-a4c9f75cba5b" containerID="0f5907168f618804bcdd0073e4a7de2da21b8f12f7bca46c5f28d9f31389f694" exitCode=0 Sep 16 14:03:25.137518 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:25.137127 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bf580f-hk8p6" event={"ID":"91df75b6-e934-4f46-840e-a4c9f75cba5b","Type":"ContainerDied","Data":"0f5907168f618804bcdd0073e4a7de2da21b8f12f7bca46c5f28d9f31389f694"} Sep 16 14:03:25.137518 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:25.137189 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bf580f-hk8p6" event={"ID":"91df75b6-e934-4f46-840e-a4c9f75cba5b","Type":"ContainerStarted","Data":"89cc5bb5a20d4cb302cb9746674c0e5ddf8c961941010b0fa06b739bf7a0b6da"} Sep 16 14:03:25.496915 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:25.496879 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:26.275863 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:26.275841 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bf580f-hk8p6" Sep 16 14:03:26.340340 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:26.340306 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:26.353667 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:26.353642 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91df75b6-e934-4f46-840e-a4c9f75cba5b-maas-api-service-ca\") pod \"91df75b6-e934-4f46-840e-a4c9f75cba5b\" (UID: \"91df75b6-e934-4f46-840e-a4c9f75cba5b\") " Sep 16 14:03:26.354016 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:26.353990 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91df75b6-e934-4f46-840e-a4c9f75cba5b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "91df75b6-e934-4f46-840e-a4c9f75cba5b" (UID: "91df75b6-e934-4f46-840e-a4c9f75cba5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:03:26.455443 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:26.455339 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91df75b6-e934-4f46-840e-a4c9f75cba5b-maas-api-service-ca\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:03:27.029252 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:27.029222 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:27.148242 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:27.148210 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bf580f-hk8p6" Sep 16 14:03:27.148391 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:27.148245 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bf580f-hk8p6" event={"ID":"91df75b6-e934-4f46-840e-a4c9f75cba5b","Type":"ContainerDied","Data":"89cc5bb5a20d4cb302cb9746674c0e5ddf8c961941010b0fa06b739bf7a0b6da"} Sep 16 14:03:27.148391 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:27.148294 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89cc5bb5a20d4cb302cb9746674c0e5ddf8c961941010b0fa06b739bf7a0b6da" Sep 16 14:03:27.724007 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:27.723977 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:28.495688 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:28.495656 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:34.827670 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:34.827636 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:34.928378 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:34.928343 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:35.031097 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:35.031069 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:35.112027 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:35.111909 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bf580f-hk8p6"] Sep 16 14:03:35.114225 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:35.114192 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-bf580f-hk8p6"] Sep 16 14:03:35.252821 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:35.252785 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:35.576801 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:35.576766 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:35.680963 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:35.680933 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:36.355359 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:36.355325 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91df75b6-e934-4f46-840e-a4c9f75cba5b" path="/var/lib/kubelet/pods/91df75b6-e934-4f46-840e-a4c9f75cba5b/volumes" Sep 16 14:03:36.413493 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:36.413457 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:37.177098 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:37.177063 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:38.049039 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:38.049007 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:38.899210 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:38.899173 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:39.605764 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:39.605721 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:03:41.790245 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:41.790215 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:41.915983 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:41.915949 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:42.109647 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:42.109538 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44"] Sep 16 14:03:42.109852 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:42.109824 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" podUID="ee2177e4-982f-488b-a1b9-25cc487b2e68" containerName="istio-proxy" containerID="cri-o://9e674b54a71c9726c39964155027648cf17e16e35fe56f7f4b5ee8c7e18a92ef" gracePeriod=30 Sep 16 14:03:42.146037 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:42.146005 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:46.087499 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:46.087423 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:46.970994 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:46.970957 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:47.255056 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.255025 2580 generic.go:356] "Generic (PLEG): container finished" podID="ee2177e4-982f-488b-a1b9-25cc487b2e68" containerID="9e674b54a71c9726c39964155027648cf17e16e35fe56f7f4b5ee8c7e18a92ef" exitCode=0 Sep 16 14:03:47.255519 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.255098 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" event={"ID":"ee2177e4-982f-488b-a1b9-25cc487b2e68","Type":"ContainerDied","Data":"9e674b54a71c9726c39964155027648cf17e16e35fe56f7f4b5ee8c7e18a92ef"} Sep 16 14:03:47.255519 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.255244 2580 scope.go:117] "RemoveContainer" containerID="c8d380af5738d3182450e0622c80d20dbe51b791eea3ad65202eb975c4d8e25a" Sep 16 14:03:47.361148 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.361116 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:03:47.465245 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.465211 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-token\") pod \"ee2177e4-982f-488b-a1b9-25cc487b2e68\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " Sep 16 14:03:47.465245 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.465247 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee2177e4-982f-488b-a1b9-25cc487b2e68-istiod-ca-cert\") pod \"ee2177e4-982f-488b-a1b9-25cc487b2e68\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " Sep 16 14:03:47.465504 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.465273 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjr9g\" (UniqueName: \"kubernetes.io/projected/ee2177e4-982f-488b-a1b9-25cc487b2e68-kube-api-access-sjr9g\") pod \"ee2177e4-982f-488b-a1b9-25cc487b2e68\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " Sep 16 14:03:47.465504 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.465292 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-data\") pod \"ee2177e4-982f-488b-a1b9-25cc487b2e68\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " Sep 16 14:03:47.465504 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.465376 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-workload-socket\") pod \"ee2177e4-982f-488b-a1b9-25cc487b2e68\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " Sep 16 14:03:47.465504 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.465405 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-podinfo\") pod \"ee2177e4-982f-488b-a1b9-25cc487b2e68\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " Sep 16 14:03:47.465504 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.465488 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-envoy\") pod \"ee2177e4-982f-488b-a1b9-25cc487b2e68\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " Sep 16 14:03:47.465792 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.465514 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-workload-certs\") pod \"ee2177e4-982f-488b-a1b9-25cc487b2e68\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " Sep 16 14:03:47.465792 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.465557 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-credential-socket\") pod \"ee2177e4-982f-488b-a1b9-25cc487b2e68\" (UID: \"ee2177e4-982f-488b-a1b9-25cc487b2e68\") " Sep 16 14:03:47.465792 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.465643 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-data" (OuterVolumeSpecName: "istio-data") pod "ee2177e4-982f-488b-a1b9-25cc487b2e68" (UID: "ee2177e4-982f-488b-a1b9-25cc487b2e68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:03:47.465792 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.465736 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee2177e4-982f-488b-a1b9-25cc487b2e68-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ee2177e4-982f-488b-a1b9-25cc487b2e68" (UID: "ee2177e4-982f-488b-a1b9-25cc487b2e68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:03:47.466000 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.465819 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ee2177e4-982f-488b-a1b9-25cc487b2e68" (UID: "ee2177e4-982f-488b-a1b9-25cc487b2e68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:03:47.466422 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.466101 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee2177e4-982f-488b-a1b9-25cc487b2e68-istiod-ca-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:03:47.466422 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.466124 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-data\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:03:47.466422 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.466138 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-workload-socket\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:03:47.466422 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.466072 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ee2177e4-982f-488b-a1b9-25cc487b2e68" (UID: "ee2177e4-982f-488b-a1b9-25cc487b2e68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:03:47.466422 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.465854 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ee2177e4-982f-488b-a1b9-25cc487b2e68" (UID: "ee2177e4-982f-488b-a1b9-25cc487b2e68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:03:47.467830 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.467807 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee2177e4-982f-488b-a1b9-25cc487b2e68-kube-api-access-sjr9g" (OuterVolumeSpecName: "kube-api-access-sjr9g") pod "ee2177e4-982f-488b-a1b9-25cc487b2e68" (UID: "ee2177e4-982f-488b-a1b9-25cc487b2e68"). InnerVolumeSpecName "kube-api-access-sjr9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:03:47.467941 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.467906 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee2177e4-982f-488b-a1b9-25cc487b2e68" (UID: "ee2177e4-982f-488b-a1b9-25cc487b2e68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:03:47.468419 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.468394 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ee2177e4-982f-488b-a1b9-25cc487b2e68" (UID: "ee2177e4-982f-488b-a1b9-25cc487b2e68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:03:47.468419 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.468410 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ee2177e4-982f-488b-a1b9-25cc487b2e68" (UID: "ee2177e4-982f-488b-a1b9-25cc487b2e68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 14:03:47.567658 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.567598 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-envoy\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:03:47.567658 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.567649 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-workload-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:03:47.567658 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.567659 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2177e4-982f-488b-a1b9-25cc487b2e68-credential-socket\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:03:47.567874 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.567667 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-token\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:03:47.567874 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.567675 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjr9g\" (UniqueName: \"kubernetes.io/projected/ee2177e4-982f-488b-a1b9-25cc487b2e68-kube-api-access-sjr9g\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:03:47.567874 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.567684 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee2177e4-982f-488b-a1b9-25cc487b2e68-istio-podinfo\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:03:47.876773 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:47.876692 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:48.261921 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:48.261894 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" Sep 16 14:03:48.262415 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:48.261941 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44" event={"ID":"ee2177e4-982f-488b-a1b9-25cc487b2e68","Type":"ContainerDied","Data":"78f98b643b699dc0a6d7a01bc64a595c82dafc624b51fcd52eb77d2a8b9e870b"} Sep 16 14:03:48.262415 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:48.262004 2580 scope.go:117] "RemoveContainer" containerID="9e674b54a71c9726c39964155027648cf17e16e35fe56f7f4b5ee8c7e18a92ef" Sep 16 14:03:48.303737 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:48.303705 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44"] Sep 16 14:03:48.313575 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:48.313548 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-bf580f-openshift-default-c846bc6bb-hng44"] Sep 16 14:03:48.354555 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:48.354511 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee2177e4-982f-488b-a1b9-25cc487b2e68" path="/var/lib/kubelet/pods/ee2177e4-982f-488b-a1b9-25cc487b2e68/volumes" Sep 16 14:03:48.938707 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:48.938671 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:49.619754 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:49.619724 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:50.349582 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:50.349543 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:52.853489 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:52.853458 2580 scope.go:117] "RemoveContainer" containerID="302206e4e9084f2ffd8fb5e5ee2b8bd2d79ff06747b0c1d5909464f8228e7756" Sep 16 14:03:52.862825 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:52.862807 2580 scope.go:117] "RemoveContainer" containerID="5393490eb9333667fe41d965163230d74c324f39638f737fdfa7d7012039e1d6" Sep 16 14:03:56.175286 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:56.175249 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:59.109827 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:03:59.109786 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:04:01.500973 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:01.500936 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:01.531178 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:01.531140 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d"] Sep 16 14:04:01.531435 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:01.531405 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d" podUID="b412d46f-77d5-49ee-bee0-d863a2a933ba" containerName="payload-processing" containerID="cri-o://b6da5a6aa85e9994c1b483b5e37dd44af23f4ab5ce8a1356b17e9fc3d99e69d2" gracePeriod=30 Sep 16 14:04:01.539667 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:01.539640 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs"] Sep 16 14:04:01.539904 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:01.539868 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs" podUID="88f9b5c1-e57d-4eae-b095-b794482d80f8" containerName="payload-pre-processing" containerID="cri-o://d37b58e986275d4eb18fe2779fb5ccdd8766bb99fe832650a2a11202cc103728" gracePeriod=30 Sep 16 14:04:01.584015 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:01.583985 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:01.799580 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:01.799544 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d" Sep 16 14:04:01.803020 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:01.802996 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b54wv\" (UniqueName: \"kubernetes.io/projected/b412d46f-77d5-49ee-bee0-d863a2a933ba-kube-api-access-b54wv\") pod \"b412d46f-77d5-49ee-bee0-d863a2a933ba\" (UID: \"b412d46f-77d5-49ee-bee0-d863a2a933ba\") " Sep 16 14:04:01.803150 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:01.803038 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b412d46f-77d5-49ee-bee0-d863a2a933ba-plugins-config-volume\") pod \"b412d46f-77d5-49ee-bee0-d863a2a933ba\" (UID: \"b412d46f-77d5-49ee-bee0-d863a2a933ba\") " Sep 16 14:04:01.803404 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:01.803382 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b412d46f-77d5-49ee-bee0-d863a2a933ba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b412d46f-77d5-49ee-bee0-d863a2a933ba" (UID: "b412d46f-77d5-49ee-bee0-d863a2a933ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:04:01.805307 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:01.805284 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b412d46f-77d5-49ee-bee0-d863a2a933ba-kube-api-access-b54wv" (OuterVolumeSpecName: "kube-api-access-b54wv") pod "b412d46f-77d5-49ee-bee0-d863a2a933ba" (UID: "b412d46f-77d5-49ee-bee0-d863a2a933ba"). InnerVolumeSpecName "kube-api-access-b54wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:04:01.819417 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:01.819379 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs" Sep 16 14:04:01.904077 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:01.904044 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88f9b5c1-e57d-4eae-b095-b794482d80f8-plugins-config-volume\") pod \"88f9b5c1-e57d-4eae-b095-b794482d80f8\" (UID: \"88f9b5c1-e57d-4eae-b095-b794482d80f8\") " Sep 16 14:04:01.904251 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:01.904091 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcqml\" (UniqueName: \"kubernetes.io/projected/88f9b5c1-e57d-4eae-b095-b794482d80f8-kube-api-access-zcqml\") pod \"88f9b5c1-e57d-4eae-b095-b794482d80f8\" (UID: \"88f9b5c1-e57d-4eae-b095-b794482d80f8\") " Sep 16 14:04:01.904295 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:01.904272 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b54wv\" (UniqueName: \"kubernetes.io/projected/b412d46f-77d5-49ee-bee0-d863a2a933ba-kube-api-access-b54wv\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:04:01.904295 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:01.904283 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b412d46f-77d5-49ee-bee0-d863a2a933ba-plugins-config-volume\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:04:01.904438 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:01.904417 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88f9b5c1-e57d-4eae-b095-b794482d80f8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "88f9b5c1-e57d-4eae-b095-b794482d80f8" (UID: "88f9b5c1-e57d-4eae-b095-b794482d80f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:04:01.906389 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:01.906368 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f9b5c1-e57d-4eae-b095-b794482d80f8-kube-api-access-zcqml" (OuterVolumeSpecName: "kube-api-access-zcqml") pod "88f9b5c1-e57d-4eae-b095-b794482d80f8" (UID: "88f9b5c1-e57d-4eae-b095-b794482d80f8"). InnerVolumeSpecName "kube-api-access-zcqml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:04:02.004732 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.004687 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88f9b5c1-e57d-4eae-b095-b794482d80f8-plugins-config-volume\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:04:02.004732 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.004728 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcqml\" (UniqueName: \"kubernetes.io/projected/88f9b5c1-e57d-4eae-b095-b794482d80f8-kube-api-access-zcqml\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:04:02.045211 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.045136 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:02.143243 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.143203 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:02.220256 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.220229 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:02.323287 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.323199 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:02.329103 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.329073 2580 generic.go:356] "Generic (PLEG): container finished" podID="b412d46f-77d5-49ee-bee0-d863a2a933ba" containerID="b6da5a6aa85e9994c1b483b5e37dd44af23f4ab5ce8a1356b17e9fc3d99e69d2" exitCode=0 Sep 16 14:04:02.329238 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.329114 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d" Sep 16 14:04:02.329238 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.329166 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d" event={"ID":"b412d46f-77d5-49ee-bee0-d863a2a933ba","Type":"ContainerDied","Data":"b6da5a6aa85e9994c1b483b5e37dd44af23f4ab5ce8a1356b17e9fc3d99e69d2"} Sep 16 14:04:02.329351 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.329244 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d" event={"ID":"b412d46f-77d5-49ee-bee0-d863a2a933ba","Type":"ContainerDied","Data":"3321251b5903bdea8a4cbf0fea0d4c5861ba062449923fe8f063165fe7e7119c"} Sep 16 14:04:02.329351 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.329267 2580 scope.go:117] "RemoveContainer" containerID="b6da5a6aa85e9994c1b483b5e37dd44af23f4ab5ce8a1356b17e9fc3d99e69d2" Sep 16 14:04:02.330763 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.330739 2580 generic.go:356] "Generic (PLEG): container finished" podID="88f9b5c1-e57d-4eae-b095-b794482d80f8" containerID="d37b58e986275d4eb18fe2779fb5ccdd8766bb99fe832650a2a11202cc103728" exitCode=0 Sep 16 14:04:02.330863 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.330775 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs" Sep 16 14:04:02.330863 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.330798 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs" event={"ID":"88f9b5c1-e57d-4eae-b095-b794482d80f8","Type":"ContainerDied","Data":"d37b58e986275d4eb18fe2779fb5ccdd8766bb99fe832650a2a11202cc103728"} Sep 16 14:04:02.330863 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.330839 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs" event={"ID":"88f9b5c1-e57d-4eae-b095-b794482d80f8","Type":"ContainerDied","Data":"cec6c44ff66f7045ca2174fa93e3f4e9377a15bd516be9910460fabfc041ee55"} Sep 16 14:04:02.338421 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.338404 2580 scope.go:117] "RemoveContainer" containerID="b6da5a6aa85e9994c1b483b5e37dd44af23f4ab5ce8a1356b17e9fc3d99e69d2" Sep 16 14:04:02.338799 ip-10-0-132-9 kubenswrapper[2580]: E0916 14:04:02.338719 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6da5a6aa85e9994c1b483b5e37dd44af23f4ab5ce8a1356b17e9fc3d99e69d2\": container with ID starting with b6da5a6aa85e9994c1b483b5e37dd44af23f4ab5ce8a1356b17e9fc3d99e69d2 not found: ID does not exist" containerID="b6da5a6aa85e9994c1b483b5e37dd44af23f4ab5ce8a1356b17e9fc3d99e69d2" Sep 16 14:04:02.338799 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.338745 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6da5a6aa85e9994c1b483b5e37dd44af23f4ab5ce8a1356b17e9fc3d99e69d2"} err="failed to get container status \"b6da5a6aa85e9994c1b483b5e37dd44af23f4ab5ce8a1356b17e9fc3d99e69d2\": rpc error: code = NotFound desc = could not find container \"b6da5a6aa85e9994c1b483b5e37dd44af23f4ab5ce8a1356b17e9fc3d99e69d2\": container with ID starting with b6da5a6aa85e9994c1b483b5e37dd44af23f4ab5ce8a1356b17e9fc3d99e69d2 not found: ID does not exist" Sep 16 14:04:02.338799 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.338762 2580 scope.go:117] "RemoveContainer" containerID="d37b58e986275d4eb18fe2779fb5ccdd8766bb99fe832650a2a11202cc103728" Sep 16 14:04:02.346770 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.346740 2580 scope.go:117] "RemoveContainer" containerID="d37b58e986275d4eb18fe2779fb5ccdd8766bb99fe832650a2a11202cc103728" Sep 16 14:04:02.347012 ip-10-0-132-9 kubenswrapper[2580]: E0916 14:04:02.346995 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37b58e986275d4eb18fe2779fb5ccdd8766bb99fe832650a2a11202cc103728\": container with ID starting with d37b58e986275d4eb18fe2779fb5ccdd8766bb99fe832650a2a11202cc103728 not found: ID does not exist" containerID="d37b58e986275d4eb18fe2779fb5ccdd8766bb99fe832650a2a11202cc103728" Sep 16 14:04:02.347062 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.347021 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37b58e986275d4eb18fe2779fb5ccdd8766bb99fe832650a2a11202cc103728"} err="failed to get container status \"d37b58e986275d4eb18fe2779fb5ccdd8766bb99fe832650a2a11202cc103728\": rpc error: code = NotFound desc = could not find container \"d37b58e986275d4eb18fe2779fb5ccdd8766bb99fe832650a2a11202cc103728\": container with ID starting with d37b58e986275d4eb18fe2779fb5ccdd8766bb99fe832650a2a11202cc103728 not found: ID does not exist" Sep 16 14:04:02.356133 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.356095 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs"] Sep 16 14:04:02.358060 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.358022 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b59887-df5cd65c4-fg4zs"] Sep 16 14:04:02.367649 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.367630 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d"] Sep 16 14:04:02.372727 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.372708 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b59887-6c5c9f74bc-8hh4d"] Sep 16 14:04:02.971187 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:02.971139 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:03.732126 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:03.732095 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:04.354211 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:04.354178 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88f9b5c1-e57d-4eae-b095-b794482d80f8" path="/var/lib/kubelet/pods/88f9b5c1-e57d-4eae-b095-b794482d80f8/volumes" Sep 16 14:04:04.354588 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:04.354580 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b412d46f-77d5-49ee-bee0-d863a2a933ba" path="/var/lib/kubelet/pods/b412d46f-77d5-49ee-bee0-d863a2a933ba/volumes" Sep 16 14:04:04.463619 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:04.463586 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:07.121942 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:07.121905 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45"] Sep 16 14:04:07.166848 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:07.166817 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:11.901255 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:11.901223 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:04:22.142175 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.142115 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" podUID="4e6d42b5-2015-4d39-8647-2fa231dcc643" containerName="main" containerID="cri-o://688c6812cc8c78cf03a6fed305de9ba9bfc1bf5fc8385a86de930d6a872a893a" gracePeriod=45 Sep 16 14:04:22.390345 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.390316 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:04:22.431468 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.431381 2580 generic.go:356] "Generic (PLEG): container finished" podID="4e6d42b5-2015-4d39-8647-2fa231dcc643" containerID="688c6812cc8c78cf03a6fed305de9ba9bfc1bf5fc8385a86de930d6a872a893a" exitCode=0 Sep 16 14:04:22.431468 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.431444 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" Sep 16 14:04:22.431735 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.431470 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" event={"ID":"4e6d42b5-2015-4d39-8647-2fa231dcc643","Type":"ContainerDied","Data":"688c6812cc8c78cf03a6fed305de9ba9bfc1bf5fc8385a86de930d6a872a893a"} Sep 16 14:04:22.431735 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.431517 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45" event={"ID":"4e6d42b5-2015-4d39-8647-2fa231dcc643","Type":"ContainerDied","Data":"00334c5b4487dd829ce22b965297ee040ccbb95ee56c24e7a90b7f7329712fb5"} Sep 16 14:04:22.431735 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.431534 2580 scope.go:117] "RemoveContainer" containerID="688c6812cc8c78cf03a6fed305de9ba9bfc1bf5fc8385a86de930d6a872a893a" Sep 16 14:04:22.442799 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.442716 2580 scope.go:117] "RemoveContainer" containerID="688c6812cc8c78cf03a6fed305de9ba9bfc1bf5fc8385a86de930d6a872a893a" Sep 16 14:04:22.443282 ip-10-0-132-9 kubenswrapper[2580]: E0916 14:04:22.443254 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"688c6812cc8c78cf03a6fed305de9ba9bfc1bf5fc8385a86de930d6a872a893a\": container with ID starting with 688c6812cc8c78cf03a6fed305de9ba9bfc1bf5fc8385a86de930d6a872a893a not found: ID does not exist" containerID="688c6812cc8c78cf03a6fed305de9ba9bfc1bf5fc8385a86de930d6a872a893a" Sep 16 14:04:22.443369 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.443298 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688c6812cc8c78cf03a6fed305de9ba9bfc1bf5fc8385a86de930d6a872a893a"} err="failed to get container status \"688c6812cc8c78cf03a6fed305de9ba9bfc1bf5fc8385a86de930d6a872a893a\": rpc error: code = NotFound desc = could not find container \"688c6812cc8c78cf03a6fed305de9ba9bfc1bf5fc8385a86de930d6a872a893a\": container with ID starting with 688c6812cc8c78cf03a6fed305de9ba9bfc1bf5fc8385a86de930d6a872a893a not found: ID does not exist" Sep 16 14:04:22.477815 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.477785 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-home\") pod \"4e6d42b5-2015-4d39-8647-2fa231dcc643\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " Sep 16 14:04:22.478002 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.477842 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-dshm\") pod \"4e6d42b5-2015-4d39-8647-2fa231dcc643\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " Sep 16 14:04:22.478002 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.477872 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-model-cache\") pod \"4e6d42b5-2015-4d39-8647-2fa231dcc643\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " Sep 16 14:04:22.478002 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.477895 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-tmp-dir\") pod \"4e6d42b5-2015-4d39-8647-2fa231dcc643\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " Sep 16 14:04:22.478002 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.477915 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e6d42b5-2015-4d39-8647-2fa231dcc643-tls-certs\") pod \"4e6d42b5-2015-4d39-8647-2fa231dcc643\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " Sep 16 14:04:22.478002 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.477981 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxwpf\" (UniqueName: \"kubernetes.io/projected/4e6d42b5-2015-4d39-8647-2fa231dcc643-kube-api-access-kxwpf\") pod \"4e6d42b5-2015-4d39-8647-2fa231dcc643\" (UID: \"4e6d42b5-2015-4d39-8647-2fa231dcc643\") " Sep 16 14:04:22.478272 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.478056 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-home" (OuterVolumeSpecName: "home") pod "4e6d42b5-2015-4d39-8647-2fa231dcc643" (UID: "4e6d42b5-2015-4d39-8647-2fa231dcc643"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:04:22.478272 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.478159 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e6d42b5-2015-4d39-8647-2fa231dcc643" (UID: "4e6d42b5-2015-4d39-8647-2fa231dcc643"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:04:22.478272 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.478169 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e6d42b5-2015-4d39-8647-2fa231dcc643" (UID: "4e6d42b5-2015-4d39-8647-2fa231dcc643"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:04:22.478393 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.478368 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:04:22.478393 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.478387 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:04:22.478474 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.478403 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:04:22.480324 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.480299 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e6d42b5-2015-4d39-8647-2fa231dcc643-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e6d42b5-2015-4d39-8647-2fa231dcc643" (UID: "4e6d42b5-2015-4d39-8647-2fa231dcc643"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:04:22.480413 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.480384 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6d42b5-2015-4d39-8647-2fa231dcc643-kube-api-access-kxwpf" (OuterVolumeSpecName: "kube-api-access-kxwpf") pod "4e6d42b5-2015-4d39-8647-2fa231dcc643" (UID: "4e6d42b5-2015-4d39-8647-2fa231dcc643"). InnerVolumeSpecName "kube-api-access-kxwpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:04:22.480564 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.480547 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-dshm" (OuterVolumeSpecName: "dshm") pod "4e6d42b5-2015-4d39-8647-2fa231dcc643" (UID: "4e6d42b5-2015-4d39-8647-2fa231dcc643"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:04:22.579626 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.579564 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxwpf\" (UniqueName: \"kubernetes.io/projected/4e6d42b5-2015-4d39-8647-2fa231dcc643-kube-api-access-kxwpf\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:04:22.579626 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.579598 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e6d42b5-2015-4d39-8647-2fa231dcc643-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:04:22.579626 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.579631 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e6d42b5-2015-4d39-8647-2fa231dcc643-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:04:22.754284 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.754253 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45"] Sep 16 14:04:22.758578 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:22.758556 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-b59887/ipp-route-b59887-kserve-67b98f8db7-w2j45"] Sep 16 14:04:24.356982 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:24.356935 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e6d42b5-2015-4d39-8647-2fa231dcc643" path="/var/lib/kubelet/pods/4e6d42b5-2015-4d39-8647-2fa231dcc643/volumes" Sep 16 14:04:43.687347 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:43.687310 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:45.472432 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:45.472397 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:50.948064 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:50.948028 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:56.911571 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:56.911529 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:57.113019 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.112980 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:57.539770 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.539735 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:57.811398 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.811317 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ac4a3c-ckwfq"] Sep 16 14:04:57.811733 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.811719 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee2177e4-982f-488b-a1b9-25cc487b2e68" containerName="istio-proxy" Sep 16 14:04:57.811791 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.811736 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee2177e4-982f-488b-a1b9-25cc487b2e68" containerName="istio-proxy" Sep 16 14:04:57.811791 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.811747 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88f9b5c1-e57d-4eae-b095-b794482d80f8" containerName="payload-pre-processing" Sep 16 14:04:57.811791 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.811753 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="88f9b5c1-e57d-4eae-b095-b794482d80f8" containerName="payload-pre-processing" Sep 16 14:04:57.811791 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.811762 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee2177e4-982f-488b-a1b9-25cc487b2e68" containerName="istio-proxy" Sep 16 14:04:57.811791 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.811767 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee2177e4-982f-488b-a1b9-25cc487b2e68" containerName="istio-proxy" Sep 16 14:04:57.811791 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.811777 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b412d46f-77d5-49ee-bee0-d863a2a933ba" containerName="payload-processing" Sep 16 14:04:57.811791 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.811782 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b412d46f-77d5-49ee-bee0-d863a2a933ba" containerName="payload-processing" Sep 16 14:04:57.811791 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.811788 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91df75b6-e934-4f46-840e-a4c9f75cba5b" containerName="revoke-keys" Sep 16 14:04:57.811791 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.811793 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="91df75b6-e934-4f46-840e-a4c9f75cba5b" containerName="revoke-keys" Sep 16 14:04:57.812072 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.811803 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e6d42b5-2015-4d39-8647-2fa231dcc643" containerName="main" Sep 16 14:04:57.812072 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.811808 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e6d42b5-2015-4d39-8647-2fa231dcc643" containerName="main" Sep 16 14:04:57.812072 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.811855 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee2177e4-982f-488b-a1b9-25cc487b2e68" containerName="istio-proxy" Sep 16 14:04:57.812072 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.811862 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e6d42b5-2015-4d39-8647-2fa231dcc643" containerName="main" Sep 16 14:04:57.812072 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.811868 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee2177e4-982f-488b-a1b9-25cc487b2e68" containerName="istio-proxy" Sep 16 14:04:57.812072 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.811875 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="88f9b5c1-e57d-4eae-b095-b794482d80f8" containerName="payload-pre-processing" Sep 16 14:04:57.812072 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.811880 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b412d46f-77d5-49ee-bee0-d863a2a933ba" containerName="payload-processing" Sep 16 14:04:57.812072 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.811885 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="91df75b6-e934-4f46-840e-a4c9f75cba5b" containerName="revoke-keys" Sep 16 14:04:57.814831 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.814815 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ac4a3c-ckwfq" Sep 16 14:04:57.817646 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.817598 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 14:04:57.817784 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.817620 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-thk5m\"" Sep 16 14:04:57.824345 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.824314 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ac4a3c-ckwfq"] Sep 16 14:04:57.901189 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:57.901152 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0227a3e7-2e92-463c-adff-eaa2c52b648b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ac4a3c-ckwfq\" (UID: \"0227a3e7-2e92-463c-adff-eaa2c52b648b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ac4a3c-ckwfq" Sep 16 14:04:58.001654 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:58.001600 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0227a3e7-2e92-463c-adff-eaa2c52b648b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ac4a3c-ckwfq\" (UID: \"0227a3e7-2e92-463c-adff-eaa2c52b648b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ac4a3c-ckwfq" Sep 16 14:04:58.002256 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:58.002235 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0227a3e7-2e92-463c-adff-eaa2c52b648b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ac4a3c-ckwfq\" (UID: \"0227a3e7-2e92-463c-adff-eaa2c52b648b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ac4a3c-ckwfq" Sep 16 14:04:58.125764 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:58.125678 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ac4a3c-ckwfq" Sep 16 14:04:58.156246 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:58.156194 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:04:58.429899 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:58.429691 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ac4a3c-ckwfq"] Sep 16 14:04:58.432540 ip-10-0-132-9 kubenswrapper[2580]: W0916 14:04:58.432509 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0227a3e7_2e92_463c_adff_eaa2c52b648b.slice/crio-af99c86fd2e6fde57887f1e98723367239ec44289e2039822c78b3f03f94deea WatchSource:0}: Error finding container af99c86fd2e6fde57887f1e98723367239ec44289e2039822c78b3f03f94deea: Status 404 returned error can't find the container with id af99c86fd2e6fde57887f1e98723367239ec44289e2039822c78b3f03f94deea Sep 16 14:04:58.597756 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:58.597721 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ac4a3c-ckwfq" event={"ID":"0227a3e7-2e92-463c-adff-eaa2c52b648b","Type":"ContainerStarted","Data":"af99c86fd2e6fde57887f1e98723367239ec44289e2039822c78b3f03f94deea"} Sep 16 14:04:59.603471 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:59.603431 2580 generic.go:356] "Generic (PLEG): container finished" podID="0227a3e7-2e92-463c-adff-eaa2c52b648b" containerID="772a1a0f755f2c7b2fdbf5554a754ed7ba967bb2c91d08e4f09e423e2811bb99" exitCode=0 Sep 16 14:04:59.603916 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:59.603469 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ac4a3c-ckwfq" event={"ID":"0227a3e7-2e92-463c-adff-eaa2c52b648b","Type":"ContainerDied","Data":"772a1a0f755f2c7b2fdbf5554a754ed7ba967bb2c91d08e4f09e423e2811bb99"} Sep 16 14:04:59.778045 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:04:59.778012 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:00.736928 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:00.736905 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ac4a3c-ckwfq" Sep 16 14:05:00.811827 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:00.811790 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:00.830907 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:00.830875 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0227a3e7-2e92-463c-adff-eaa2c52b648b-maas-api-service-ca\") pod \"0227a3e7-2e92-463c-adff-eaa2c52b648b\" (UID: \"0227a3e7-2e92-463c-adff-eaa2c52b648b\") " Sep 16 14:05:00.831227 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:00.831204 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0227a3e7-2e92-463c-adff-eaa2c52b648b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0227a3e7-2e92-463c-adff-eaa2c52b648b" (UID: "0227a3e7-2e92-463c-adff-eaa2c52b648b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:05:00.932209 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:00.932118 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0227a3e7-2e92-463c-adff-eaa2c52b648b-maas-api-service-ca\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:05:01.614726 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:01.614695 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ac4a3c-ckwfq" event={"ID":"0227a3e7-2e92-463c-adff-eaa2c52b648b","Type":"ContainerDied","Data":"af99c86fd2e6fde57887f1e98723367239ec44289e2039822c78b3f03f94deea"} Sep 16 14:05:01.614726 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:01.614723 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ac4a3c-ckwfq" Sep 16 14:05:01.614726 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:01.614741 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af99c86fd2e6fde57887f1e98723367239ec44289e2039822c78b3f03f94deea" Sep 16 14:05:01.790015 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:01.789977 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:02.778396 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:02.778348 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:08.526299 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:08.526265 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:08.643030 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:08.642993 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:08.723944 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:08.723914 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:08.798173 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:08.798085 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ac4a3c-ckwfq"] Sep 16 14:05:08.801275 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:08.801246 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ac4a3c-ckwfq"] Sep 16 14:05:08.855250 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:08.855219 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:08.939490 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:08.939458 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:09.249004 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:09.248972 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:09.376214 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:09.376184 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:10.334954 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:10.334921 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:10.355288 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:10.355258 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0227a3e7-2e92-463c-adff-eaa2c52b648b" path="/var/lib/kubelet/pods/0227a3e7-2e92-463c-adff-eaa2c52b648b/volumes" Sep 16 14:05:11.529558 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:11.529523 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:11.910410 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:11.910321 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:05:14.339245 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:14.339168 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:15.440306 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:15.440269 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:15.538266 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:15.538232 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:15.692836 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:15.692748 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845"] Sep 16 14:05:15.693143 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:15.693114 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" podUID="a8d1c586-e589-4690-a507-98f8f26c5c42" containerName="istio-proxy" containerID="cri-o://60f66969db0e25fe603e91b1e8021810617938346d8bbbb263f408a12f8e1e3f" gracePeriod=30 Sep 16 14:05:15.721012 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:15.720977 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:17.086307 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:17.086269 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:17.406880 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:17.406769 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:18.355617 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:18.355567 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:19.517416 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:19.517381 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:20.947168 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:20.947142 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:05:21.125356 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.125327 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2df5\" (UniqueName: \"kubernetes.io/projected/a8d1c586-e589-4690-a507-98f8f26c5c42-kube-api-access-f2df5\") pod \"a8d1c586-e589-4690-a507-98f8f26c5c42\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " Sep 16 14:05:21.125561 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.125369 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-token\") pod \"a8d1c586-e589-4690-a507-98f8f26c5c42\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " Sep 16 14:05:21.125561 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.125396 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d1c586-e589-4690-a507-98f8f26c5c42-istiod-ca-cert\") pod \"a8d1c586-e589-4690-a507-98f8f26c5c42\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " Sep 16 14:05:21.125561 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.125421 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-envoy\") pod \"a8d1c586-e589-4690-a507-98f8f26c5c42\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " Sep 16 14:05:21.125561 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.125446 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-workload-socket\") pod \"a8d1c586-e589-4690-a507-98f8f26c5c42\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " Sep 16 14:05:21.125561 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.125479 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-workload-certs\") pod \"a8d1c586-e589-4690-a507-98f8f26c5c42\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " Sep 16 14:05:21.125561 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.125505 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-podinfo\") pod \"a8d1c586-e589-4690-a507-98f8f26c5c42\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " Sep 16 14:05:21.125561 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.125548 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-data\") pod \"a8d1c586-e589-4690-a507-98f8f26c5c42\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " Sep 16 14:05:21.125990 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.125583 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-credential-socket\") pod \"a8d1c586-e589-4690-a507-98f8f26c5c42\" (UID: \"a8d1c586-e589-4690-a507-98f8f26c5c42\") " Sep 16 14:05:21.125990 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.125845 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a8d1c586-e589-4690-a507-98f8f26c5c42" (UID: "a8d1c586-e589-4690-a507-98f8f26c5c42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:05:21.125990 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.125862 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a8d1c586-e589-4690-a507-98f8f26c5c42" (UID: "a8d1c586-e589-4690-a507-98f8f26c5c42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:05:21.126337 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.126028 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a8d1c586-e589-4690-a507-98f8f26c5c42" (UID: "a8d1c586-e589-4690-a507-98f8f26c5c42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:05:21.126337 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.126041 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-data" (OuterVolumeSpecName: "istio-data") pod "a8d1c586-e589-4690-a507-98f8f26c5c42" (UID: "a8d1c586-e589-4690-a507-98f8f26c5c42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:05:21.126337 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.126147 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d1c586-e589-4690-a507-98f8f26c5c42-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a8d1c586-e589-4690-a507-98f8f26c5c42" (UID: "a8d1c586-e589-4690-a507-98f8f26c5c42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:05:21.127892 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.127867 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-token" (OuterVolumeSpecName: "istio-token") pod "a8d1c586-e589-4690-a507-98f8f26c5c42" (UID: "a8d1c586-e589-4690-a507-98f8f26c5c42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:05:21.128015 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.127948 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a8d1c586-e589-4690-a507-98f8f26c5c42" (UID: "a8d1c586-e589-4690-a507-98f8f26c5c42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 14:05:21.128015 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.127963 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d1c586-e589-4690-a507-98f8f26c5c42-kube-api-access-f2df5" (OuterVolumeSpecName: "kube-api-access-f2df5") pod "a8d1c586-e589-4690-a507-98f8f26c5c42" (UID: "a8d1c586-e589-4690-a507-98f8f26c5c42"). InnerVolumeSpecName "kube-api-access-f2df5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:05:21.128015 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.127982 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a8d1c586-e589-4690-a507-98f8f26c5c42" (UID: "a8d1c586-e589-4690-a507-98f8f26c5c42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:05:21.226669 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.226630 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-workload-socket\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:05:21.226669 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.226663 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-workload-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:05:21.226669 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.226677 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-podinfo\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:05:21.226921 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.226688 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-data\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:05:21.226921 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.226699 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-credential-socket\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:05:21.226921 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.226710 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2df5\" (UniqueName: \"kubernetes.io/projected/a8d1c586-e589-4690-a507-98f8f26c5c42-kube-api-access-f2df5\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:05:21.226921 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.226721 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-token\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:05:21.226921 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.226732 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d1c586-e589-4690-a507-98f8f26c5c42-istiod-ca-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:05:21.226921 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.226743 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d1c586-e589-4690-a507-98f8f26c5c42-istio-envoy\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:05:21.711191 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.711154 2580 generic.go:356] "Generic (PLEG): container finished" podID="a8d1c586-e589-4690-a507-98f8f26c5c42" containerID="60f66969db0e25fe603e91b1e8021810617938346d8bbbb263f408a12f8e1e3f" exitCode=0 Sep 16 14:05:21.711361 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.711214 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" Sep 16 14:05:21.711361 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.711237 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" event={"ID":"a8d1c586-e589-4690-a507-98f8f26c5c42","Type":"ContainerDied","Data":"60f66969db0e25fe603e91b1e8021810617938346d8bbbb263f408a12f8e1e3f"} Sep 16 14:05:21.711361 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.711295 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845" event={"ID":"a8d1c586-e589-4690-a507-98f8f26c5c42","Type":"ContainerDied","Data":"d104dcf703dfc48696d5a0c64509bdb03d86898edcc8d6b20ead9be20e9a509b"} Sep 16 14:05:21.711361 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.711316 2580 scope.go:117] "RemoveContainer" containerID="60f66969db0e25fe603e91b1e8021810617938346d8bbbb263f408a12f8e1e3f" Sep 16 14:05:21.722724 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.722708 2580 scope.go:117] "RemoveContainer" containerID="30211464e3d639197bd9e5ebed67552628061067a20f7dad87161392b33732d8" Sep 16 14:05:21.730638 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.730617 2580 scope.go:117] "RemoveContainer" containerID="60f66969db0e25fe603e91b1e8021810617938346d8bbbb263f408a12f8e1e3f" Sep 16 14:05:21.730886 ip-10-0-132-9 kubenswrapper[2580]: E0916 14:05:21.730867 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f66969db0e25fe603e91b1e8021810617938346d8bbbb263f408a12f8e1e3f\": container with ID starting with 60f66969db0e25fe603e91b1e8021810617938346d8bbbb263f408a12f8e1e3f not found: ID does not exist" containerID="60f66969db0e25fe603e91b1e8021810617938346d8bbbb263f408a12f8e1e3f" Sep 16 14:05:21.730926 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.730894 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f66969db0e25fe603e91b1e8021810617938346d8bbbb263f408a12f8e1e3f"} err="failed to get container status \"60f66969db0e25fe603e91b1e8021810617938346d8bbbb263f408a12f8e1e3f\": rpc error: code = NotFound desc = could not find container \"60f66969db0e25fe603e91b1e8021810617938346d8bbbb263f408a12f8e1e3f\": container with ID starting with 60f66969db0e25fe603e91b1e8021810617938346d8bbbb263f408a12f8e1e3f not found: ID does not exist" Sep 16 14:05:21.730926 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.730911 2580 scope.go:117] "RemoveContainer" containerID="30211464e3d639197bd9e5ebed67552628061067a20f7dad87161392b33732d8" Sep 16 14:05:21.731148 ip-10-0-132-9 kubenswrapper[2580]: E0916 14:05:21.731133 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30211464e3d639197bd9e5ebed67552628061067a20f7dad87161392b33732d8\": container with ID starting with 30211464e3d639197bd9e5ebed67552628061067a20f7dad87161392b33732d8 not found: ID does not exist" containerID="30211464e3d639197bd9e5ebed67552628061067a20f7dad87161392b33732d8" Sep 16 14:05:21.731184 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.731152 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30211464e3d639197bd9e5ebed67552628061067a20f7dad87161392b33732d8"} err="failed to get container status \"30211464e3d639197bd9e5ebed67552628061067a20f7dad87161392b33732d8\": rpc error: code = NotFound desc = could not find container \"30211464e3d639197bd9e5ebed67552628061067a20f7dad87161392b33732d8\": container with ID starting with 30211464e3d639197bd9e5ebed67552628061067a20f7dad87161392b33732d8 not found: ID does not exist" Sep 16 14:05:21.735977 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.735952 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845"] Sep 16 14:05:21.743514 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:21.743493 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ac4a3c-openshift-default-5499cb4fdb-ts845"] Sep 16 14:05:22.355445 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:22.355412 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8d1c586-e589-4690-a507-98f8f26c5c42" path="/var/lib/kubelet/pods/a8d1c586-e589-4690-a507-98f8f26c5c42/volumes" Sep 16 14:05:26.819185 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:26.819147 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:26.850572 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:26.850536 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v"] Sep 16 14:05:26.850795 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:26.850767 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v" podUID="59c23cb2-d0f4-4467-a8a3-2842e3d30cf3" containerName="payload-processing" containerID="cri-o://1b4f1092dfb4eabf00111bc29d2ede458d0e3160b44c90846d6806e2d0e53f12" gracePeriod=30 Sep 16 14:05:26.924064 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:26.924036 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:27.035026 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:27.034999 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:27.104375 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:27.104352 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v" Sep 16 14:05:27.138422 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:27.138394 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:27.179835 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:27.179805 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59c23cb2-d0f4-4467-a8a3-2842e3d30cf3-plugins-config-volume\") pod \"59c23cb2-d0f4-4467-a8a3-2842e3d30cf3\" (UID: \"59c23cb2-d0f4-4467-a8a3-2842e3d30cf3\") " Sep 16 14:05:27.180005 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:27.179875 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvbz4\" (UniqueName: \"kubernetes.io/projected/59c23cb2-d0f4-4467-a8a3-2842e3d30cf3-kube-api-access-dvbz4\") pod \"59c23cb2-d0f4-4467-a8a3-2842e3d30cf3\" (UID: \"59c23cb2-d0f4-4467-a8a3-2842e3d30cf3\") " Sep 16 14:05:27.180287 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:27.180258 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c23cb2-d0f4-4467-a8a3-2842e3d30cf3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "59c23cb2-d0f4-4467-a8a3-2842e3d30cf3" (UID: "59c23cb2-d0f4-4467-a8a3-2842e3d30cf3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:05:27.182249 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:27.182221 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c23cb2-d0f4-4467-a8a3-2842e3d30cf3-kube-api-access-dvbz4" (OuterVolumeSpecName: "kube-api-access-dvbz4") pod "59c23cb2-d0f4-4467-a8a3-2842e3d30cf3" (UID: "59c23cb2-d0f4-4467-a8a3-2842e3d30cf3"). InnerVolumeSpecName "kube-api-access-dvbz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:05:27.256880 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:27.256848 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:27.281312 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:27.281282 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59c23cb2-d0f4-4467-a8a3-2842e3d30cf3-plugins-config-volume\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:05:27.281312 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:27.281308 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvbz4\" (UniqueName: \"kubernetes.io/projected/59c23cb2-d0f4-4467-a8a3-2842e3d30cf3-kube-api-access-dvbz4\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:05:27.579163 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:27.579126 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:27.701845 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:27.701813 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:27.741365 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:27.741335 2580 generic.go:356] "Generic (PLEG): container finished" podID="59c23cb2-d0f4-4467-a8a3-2842e3d30cf3" containerID="1b4f1092dfb4eabf00111bc29d2ede458d0e3160b44c90846d6806e2d0e53f12" exitCode=0 Sep 16 14:05:27.741544 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:27.741378 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v" Sep 16 14:05:27.741544 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:27.741426 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v" event={"ID":"59c23cb2-d0f4-4467-a8a3-2842e3d30cf3","Type":"ContainerDied","Data":"1b4f1092dfb4eabf00111bc29d2ede458d0e3160b44c90846d6806e2d0e53f12"} Sep 16 14:05:27.741544 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:27.741486 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v" event={"ID":"59c23cb2-d0f4-4467-a8a3-2842e3d30cf3","Type":"ContainerDied","Data":"ee25cd77db016bce4c24a49b4e42f1947109994c4ad37e270ebbe6230ca5e05f"} Sep 16 14:05:27.741544 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:27.741516 2580 scope.go:117] "RemoveContainer" containerID="1b4f1092dfb4eabf00111bc29d2ede458d0e3160b44c90846d6806e2d0e53f12" Sep 16 14:05:27.750355 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:27.750335 2580 scope.go:117] "RemoveContainer" containerID="1b4f1092dfb4eabf00111bc29d2ede458d0e3160b44c90846d6806e2d0e53f12" Sep 16 14:05:27.750655 ip-10-0-132-9 kubenswrapper[2580]: E0916 14:05:27.750635 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b4f1092dfb4eabf00111bc29d2ede458d0e3160b44c90846d6806e2d0e53f12\": container with ID starting with 1b4f1092dfb4eabf00111bc29d2ede458d0e3160b44c90846d6806e2d0e53f12 not found: ID does not exist" containerID="1b4f1092dfb4eabf00111bc29d2ede458d0e3160b44c90846d6806e2d0e53f12" Sep 16 14:05:27.750735 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:27.750662 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b4f1092dfb4eabf00111bc29d2ede458d0e3160b44c90846d6806e2d0e53f12"} err="failed to get container status \"1b4f1092dfb4eabf00111bc29d2ede458d0e3160b44c90846d6806e2d0e53f12\": rpc error: code = NotFound desc = could not find container \"1b4f1092dfb4eabf00111bc29d2ede458d0e3160b44c90846d6806e2d0e53f12\": container with ID starting with 1b4f1092dfb4eabf00111bc29d2ede458d0e3160b44c90846d6806e2d0e53f12 not found: ID does not exist" Sep 16 14:05:27.770077 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:27.770049 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v"] Sep 16 14:05:27.774234 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:27.774212 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-973610-5866764647-pss5v"] Sep 16 14:05:28.355798 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:28.355754 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59c23cb2-d0f4-4467-a8a3-2842e3d30cf3" path="/var/lib/kubelet/pods/59c23cb2-d0f4-4467-a8a3-2842e3d30cf3/volumes" Sep 16 14:05:28.664054 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:28.663959 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:30.039428 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:30.039391 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:05:33.995502 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:33.995436 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:34.098559 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.098519 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:34.306119 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.306044 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:34.653006 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.652930 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c36a6-l5zz9"] Sep 16 14:05:34.653255 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.653242 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c23cb2-d0f4-4467-a8a3-2842e3d30cf3" containerName="payload-processing" Sep 16 14:05:34.653301 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.653255 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c23cb2-d0f4-4467-a8a3-2842e3d30cf3" containerName="payload-processing" Sep 16 14:05:34.653301 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.653270 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0227a3e7-2e92-463c-adff-eaa2c52b648b" containerName="revoke-keys" Sep 16 14:05:34.653301 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.653275 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0227a3e7-2e92-463c-adff-eaa2c52b648b" containerName="revoke-keys" Sep 16 14:05:34.653301 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.653284 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8d1c586-e589-4690-a507-98f8f26c5c42" containerName="istio-proxy" Sep 16 14:05:34.653301 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.653289 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8d1c586-e589-4690-a507-98f8f26c5c42" containerName="istio-proxy" Sep 16 14:05:34.653301 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.653301 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8d1c586-e589-4690-a507-98f8f26c5c42" containerName="istio-proxy" Sep 16 14:05:34.653513 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.653307 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8d1c586-e589-4690-a507-98f8f26c5c42" containerName="istio-proxy" Sep 16 14:05:34.653513 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.653360 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8d1c586-e589-4690-a507-98f8f26c5c42" containerName="istio-proxy" Sep 16 14:05:34.653513 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.653367 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0227a3e7-2e92-463c-adff-eaa2c52b648b" containerName="revoke-keys" Sep 16 14:05:34.653513 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.653374 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="59c23cb2-d0f4-4467-a8a3-2842e3d30cf3" containerName="payload-processing" Sep 16 14:05:34.653513 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.653380 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8d1c586-e589-4690-a507-98f8f26c5c42" containerName="istio-proxy" Sep 16 14:05:34.657590 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.657566 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c36a6-l5zz9" Sep 16 14:05:34.660673 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.660651 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-thk5m\"" Sep 16 14:05:34.660828 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.660809 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 14:05:34.668373 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.668344 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c36a6-l5zz9"] Sep 16 14:05:34.744515 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.744462 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c02d88-9bbc-4fea-93c3-9c73ea1c5768-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8c36a6-l5zz9\" (UID: \"15c02d88-9bbc-4fea-93c3-9c73ea1c5768\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c36a6-l5zz9" Sep 16 14:05:34.796717 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.796681 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:34.845280 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.845244 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c02d88-9bbc-4fea-93c3-9c73ea1c5768-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8c36a6-l5zz9\" (UID: \"15c02d88-9bbc-4fea-93c3-9c73ea1c5768\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c36a6-l5zz9" Sep 16 14:05:34.845938 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.845920 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c02d88-9bbc-4fea-93c3-9c73ea1c5768-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8c36a6-l5zz9\" (UID: \"15c02d88-9bbc-4fea-93c3-9c73ea1c5768\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c36a6-l5zz9" Sep 16 14:05:34.967897 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:34.967869 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c36a6-l5zz9" Sep 16 14:05:35.246543 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:35.246516 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c36a6-l5zz9"] Sep 16 14:05:35.248449 ip-10-0-132-9 kubenswrapper[2580]: W0916 14:05:35.248423 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c02d88_9bbc_4fea_93c3_9c73ea1c5768.slice/crio-3cc055010ff8ea4d38f2c915b701b8f0f4bf960c66086aa8357fd664a69ef287 WatchSource:0}: Error finding container 3cc055010ff8ea4d38f2c915b701b8f0f4bf960c66086aa8357fd664a69ef287: Status 404 returned error can't find the container with id 3cc055010ff8ea4d38f2c915b701b8f0f4bf960c66086aa8357fd664a69ef287 Sep 16 14:05:35.781018 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:35.780979 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c36a6-l5zz9" event={"ID":"15c02d88-9bbc-4fea-93c3-9c73ea1c5768","Type":"ContainerStarted","Data":"a95a6acb218f668b9d72748b4587a028716383c41cf9c28a47563c8bedeca5e3"} Sep 16 14:05:35.781213 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:35.781089 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c36a6-l5zz9" event={"ID":"15c02d88-9bbc-4fea-93c3-9c73ea1c5768","Type":"ContainerStarted","Data":"3cc055010ff8ea4d38f2c915b701b8f0f4bf960c66086aa8357fd664a69ef287"} Sep 16 14:05:35.806667 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:35.806589 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c36a6-l5zz9" podStartSLOduration=1.806572718 podStartE2EDuration="1.806572718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:05:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:05:35.803646981 +0000 UTC m=+2083.904459695" watchObservedRunningTime="2026-09-16 14:05:35.806572718 +0000 UTC m=+2083.907385432" Sep 16 14:05:36.787285 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:36.787249 2580 generic.go:356] "Generic (PLEG): container finished" podID="15c02d88-9bbc-4fea-93c3-9c73ea1c5768" containerID="a95a6acb218f668b9d72748b4587a028716383c41cf9c28a47563c8bedeca5e3" exitCode=0 Sep 16 14:05:36.787708 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:36.787338 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c36a6-l5zz9" event={"ID":"15c02d88-9bbc-4fea-93c3-9c73ea1c5768","Type":"ContainerDied","Data":"a95a6acb218f668b9d72748b4587a028716383c41cf9c28a47563c8bedeca5e3"} Sep 16 14:05:37.923453 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:37.923424 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c36a6-l5zz9" Sep 16 14:05:37.972274 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:37.972242 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c02d88-9bbc-4fea-93c3-9c73ea1c5768-maas-api-service-ca\") pod \"15c02d88-9bbc-4fea-93c3-9c73ea1c5768\" (UID: \"15c02d88-9bbc-4fea-93c3-9c73ea1c5768\") " Sep 16 14:05:37.972576 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:37.972555 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c02d88-9bbc-4fea-93c3-9c73ea1c5768-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "15c02d88-9bbc-4fea-93c3-9c73ea1c5768" (UID: "15c02d88-9bbc-4fea-93c3-9c73ea1c5768"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:05:38.073276 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:38.073200 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c02d88-9bbc-4fea-93c3-9c73ea1c5768-maas-api-service-ca\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:05:38.388820 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:38.388792 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:38.488807 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:38.488777 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:38.798517 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:38.798486 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c36a6-l5zz9" Sep 16 14:05:38.798718 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:38.798487 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c36a6-l5zz9" event={"ID":"15c02d88-9bbc-4fea-93c3-9c73ea1c5768","Type":"ContainerDied","Data":"3cc055010ff8ea4d38f2c915b701b8f0f4bf960c66086aa8357fd664a69ef287"} Sep 16 14:05:38.798718 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:38.798644 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cc055010ff8ea4d38f2c915b701b8f0f4bf960c66086aa8357fd664a69ef287" Sep 16 14:05:42.670504 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:42.670468 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:45.693183 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:45.693147 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:45.739784 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:45.739754 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c36a6-29826120-fnd8j"] Sep 16 14:05:45.751530 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:45.747059 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c36a6-29826120-fnd8j"] Sep 16 14:05:45.957591 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:45.957496 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:46.235043 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:46.235010 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:46.316951 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:46.316915 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c36a6-l5zz9"] Sep 16 14:05:46.320732 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:46.320706 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c36a6-l5zz9"] Sep 16 14:05:46.354987 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:46.354957 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15c02d88-9bbc-4fea-93c3-9c73ea1c5768" path="/var/lib/kubelet/pods/15c02d88-9bbc-4fea-93c3-9c73ea1c5768/volumes" Sep 16 14:05:46.355234 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:46.355221 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6562cc2c-077d-4cd4-b4af-18cac0a8020b" path="/var/lib/kubelet/pods/6562cc2c-077d-4cd4-b4af-18cac0a8020b/volumes" Sep 16 14:05:46.355802 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:46.355784 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:46.726773 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:46.726742 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:47.041552 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:47.041467 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:48.641136 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:48.641101 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:05:54.990365 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:54.990331 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:55.380644 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:55.380543 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9"] Sep 16 14:05:55.380795 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:55.380752 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" podUID="2698b1b4-3481-41fa-8afc-79551dee1bc8" containerName="istio-proxy" containerID="cri-o://7af7bc12b61dd0419da45081202cc0e18088f442ee08c1ba5e97a479d395c215" gracePeriod=30 Sep 16 14:05:57.773868 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:57.773832 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:05:57.872095 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:05:57.872068 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:00.232773 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.232727 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 14:06:00.233148 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.232800 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" podUID="2698b1b4-3481-41fa-8afc-79551dee1bc8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 14:06:00.631970 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.631947 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 14:06:00.782798 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.782721 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2698b1b4-3481-41fa-8afc-79551dee1bc8-istiod-ca-cert\") pod \"2698b1b4-3481-41fa-8afc-79551dee1bc8\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " Sep 16 14:06:00.782798 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.782768 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-credential-socket\") pod \"2698b1b4-3481-41fa-8afc-79551dee1bc8\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " Sep 16 14:06:00.782798 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.782790 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-envoy\") pod \"2698b1b4-3481-41fa-8afc-79551dee1bc8\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " Sep 16 14:06:00.783008 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.782933 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-data\") pod \"2698b1b4-3481-41fa-8afc-79551dee1bc8\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " Sep 16 14:06:00.783008 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.782979 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgrng\" (UniqueName: \"kubernetes.io/projected/2698b1b4-3481-41fa-8afc-79551dee1bc8-kube-api-access-rgrng\") pod \"2698b1b4-3481-41fa-8afc-79551dee1bc8\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " Sep 16 14:06:00.783078 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.783013 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-workload-certs\") pod \"2698b1b4-3481-41fa-8afc-79551dee1bc8\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " Sep 16 14:06:00.783078 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.783055 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-workload-socket\") pod \"2698b1b4-3481-41fa-8afc-79551dee1bc8\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " Sep 16 14:06:00.783078 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.783070 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2698b1b4-3481-41fa-8afc-79551dee1bc8" (UID: "2698b1b4-3481-41fa-8afc-79551dee1bc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:00.783230 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.783080 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-token\") pod \"2698b1b4-3481-41fa-8afc-79551dee1bc8\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " Sep 16 14:06:00.783230 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.783100 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2698b1b4-3481-41fa-8afc-79551dee1bc8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2698b1b4-3481-41fa-8afc-79551dee1bc8" (UID: "2698b1b4-3481-41fa-8afc-79551dee1bc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:06:00.783230 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.783112 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-podinfo\") pod \"2698b1b4-3481-41fa-8afc-79551dee1bc8\" (UID: \"2698b1b4-3481-41fa-8afc-79551dee1bc8\") " Sep 16 14:06:00.783230 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.783173 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-data" (OuterVolumeSpecName: "istio-data") pod "2698b1b4-3481-41fa-8afc-79551dee1bc8" (UID: "2698b1b4-3481-41fa-8afc-79551dee1bc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:00.783453 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.783308 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2698b1b4-3481-41fa-8afc-79551dee1bc8" (UID: "2698b1b4-3481-41fa-8afc-79551dee1bc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:00.783453 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.783362 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2698b1b4-3481-41fa-8afc-79551dee1bc8" (UID: "2698b1b4-3481-41fa-8afc-79551dee1bc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:00.783575 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.783456 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2698b1b4-3481-41fa-8afc-79551dee1bc8-istiod-ca-cert\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:06:00.783575 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.783477 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-credential-socket\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:06:00.783575 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.783491 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-data\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:06:00.783575 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.783505 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-workload-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:06:00.783575 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.783517 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-workload-socket\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:06:00.785331 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.785287 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-token" (OuterVolumeSpecName: "istio-token") pod "2698b1b4-3481-41fa-8afc-79551dee1bc8" (UID: "2698b1b4-3481-41fa-8afc-79551dee1bc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:06:00.785450 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.785376 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2698b1b4-3481-41fa-8afc-79551dee1bc8" (UID: "2698b1b4-3481-41fa-8afc-79551dee1bc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 14:06:00.785661 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.785643 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2698b1b4-3481-41fa-8afc-79551dee1bc8-kube-api-access-rgrng" (OuterVolumeSpecName: "kube-api-access-rgrng") pod "2698b1b4-3481-41fa-8afc-79551dee1bc8" (UID: "2698b1b4-3481-41fa-8afc-79551dee1bc8"). InnerVolumeSpecName "kube-api-access-rgrng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:06:00.785731 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.785660 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2698b1b4-3481-41fa-8afc-79551dee1bc8" (UID: "2698b1b4-3481-41fa-8afc-79551dee1bc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:00.884278 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.884239 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-envoy\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:06:00.884278 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.884273 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgrng\" (UniqueName: \"kubernetes.io/projected/2698b1b4-3481-41fa-8afc-79551dee1bc8-kube-api-access-rgrng\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:06:00.884278 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.884287 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-token\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:06:00.884524 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.884298 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2698b1b4-3481-41fa-8afc-79551dee1bc8-istio-podinfo\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:06:00.908438 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.908402 2580 generic.go:356] "Generic (PLEG): container finished" podID="2698b1b4-3481-41fa-8afc-79551dee1bc8" containerID="7af7bc12b61dd0419da45081202cc0e18088f442ee08c1ba5e97a479d395c215" exitCode=0 Sep 16 14:06:00.908600 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.908451 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" Sep 16 14:06:00.908600 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.908476 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" event={"ID":"2698b1b4-3481-41fa-8afc-79551dee1bc8","Type":"ContainerDied","Data":"7af7bc12b61dd0419da45081202cc0e18088f442ee08c1ba5e97a479d395c215"} Sep 16 14:06:00.908600 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.908520 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9" event={"ID":"2698b1b4-3481-41fa-8afc-79551dee1bc8","Type":"ContainerDied","Data":"a70278d6c7ac9afba03731d62439bcd136c7937fe220ed053346a6dd49609bf7"} Sep 16 14:06:00.908600 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.908542 2580 scope.go:117] "RemoveContainer" containerID="7af7bc12b61dd0419da45081202cc0e18088f442ee08c1ba5e97a479d395c215" Sep 16 14:06:00.918061 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.918039 2580 scope.go:117] "RemoveContainer" containerID="cbfb5fc428aee7bccc6c3643b2edd0d534e9764259a15ee95049ce6e627a5216" Sep 16 14:06:00.926088 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.926066 2580 scope.go:117] "RemoveContainer" containerID="7af7bc12b61dd0419da45081202cc0e18088f442ee08c1ba5e97a479d395c215" Sep 16 14:06:00.926369 ip-10-0-132-9 kubenswrapper[2580]: E0916 14:06:00.926349 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af7bc12b61dd0419da45081202cc0e18088f442ee08c1ba5e97a479d395c215\": container with ID starting with 7af7bc12b61dd0419da45081202cc0e18088f442ee08c1ba5e97a479d395c215 not found: ID does not exist" containerID="7af7bc12b61dd0419da45081202cc0e18088f442ee08c1ba5e97a479d395c215" Sep 16 14:06:00.926434 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.926378 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af7bc12b61dd0419da45081202cc0e18088f442ee08c1ba5e97a479d395c215"} err="failed to get container status \"7af7bc12b61dd0419da45081202cc0e18088f442ee08c1ba5e97a479d395c215\": rpc error: code = NotFound desc = could not find container \"7af7bc12b61dd0419da45081202cc0e18088f442ee08c1ba5e97a479d395c215\": container with ID starting with 7af7bc12b61dd0419da45081202cc0e18088f442ee08c1ba5e97a479d395c215 not found: ID does not exist" Sep 16 14:06:00.926434 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.926396 2580 scope.go:117] "RemoveContainer" containerID="cbfb5fc428aee7bccc6c3643b2edd0d534e9764259a15ee95049ce6e627a5216" Sep 16 14:06:00.926641 ip-10-0-132-9 kubenswrapper[2580]: E0916 14:06:00.926622 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbfb5fc428aee7bccc6c3643b2edd0d534e9764259a15ee95049ce6e627a5216\": container with ID starting with cbfb5fc428aee7bccc6c3643b2edd0d534e9764259a15ee95049ce6e627a5216 not found: ID does not exist" containerID="cbfb5fc428aee7bccc6c3643b2edd0d534e9764259a15ee95049ce6e627a5216" Sep 16 14:06:00.926701 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.926646 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbfb5fc428aee7bccc6c3643b2edd0d534e9764259a15ee95049ce6e627a5216"} err="failed to get container status \"cbfb5fc428aee7bccc6c3643b2edd0d534e9764259a15ee95049ce6e627a5216\": rpc error: code = NotFound desc = could not find container \"cbfb5fc428aee7bccc6c3643b2edd0d534e9764259a15ee95049ce6e627a5216\": container with ID starting with cbfb5fc428aee7bccc6c3643b2edd0d534e9764259a15ee95049ce6e627a5216 not found: ID does not exist" Sep 16 14:06:00.940529 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.940496 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9"] Sep 16 14:06:00.947768 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:00.947742 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8c36a6-openshift-default-77c8678c96-n9vq9"] Sep 16 14:06:01.544974 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:01.544943 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:06:02.360133 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:02.360102 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2698b1b4-3481-41fa-8afc-79551dee1bc8" path="/var/lib/kubelet/pods/2698b1b4-3481-41fa-8afc-79551dee1bc8/volumes" Sep 16 14:06:17.755321 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:17.755287 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:06:52.922798 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:06:52.922719 2580 scope.go:117] "RemoveContainer" containerID="241bc7e24d968ce28ba37c1df7d8dfc1a4b19428ea553b39d6c8487962f5e0bf" Sep 16 14:07:19.712573 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:07:19.712525 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:19.789465 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:07:19.789433 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:25.776799 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:07:25.776751 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:26.110241 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:07:26.110150 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:07:27.451895 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:07:27.451858 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:54.741639 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:07:54.741589 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:55.528960 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:07:55.528926 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:07:55.720514 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:07:55.720480 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:07:57.026470 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:07:57.026436 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:08:04.000103 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:08:04.000063 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:08:12.000574 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:08:12.000535 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:08:29.911553 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:08:29.911506 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:08:36.601023 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:08:36.600985 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:09:52.963368 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:09:52.963292 2580 scope.go:117] "RemoveContainer" containerID="0f5907168f618804bcdd0073e4a7de2da21b8f12f7bca46c5f28d9f31389f694" Sep 16 14:10:11.007094 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:11.007056 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:10:17.187932 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.187895 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx"] Sep 16 14:10:17.188295 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.188249 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2698b1b4-3481-41fa-8afc-79551dee1bc8" containerName="istio-proxy" Sep 16 14:10:17.188295 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.188260 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2698b1b4-3481-41fa-8afc-79551dee1bc8" containerName="istio-proxy" Sep 16 14:10:17.188295 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.188268 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2698b1b4-3481-41fa-8afc-79551dee1bc8" containerName="istio-proxy" Sep 16 14:10:17.188295 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.188274 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2698b1b4-3481-41fa-8afc-79551dee1bc8" containerName="istio-proxy" Sep 16 14:10:17.188295 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.188280 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15c02d88-9bbc-4fea-93c3-9c73ea1c5768" containerName="revoke-keys" Sep 16 14:10:17.188295 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.188286 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15c02d88-9bbc-4fea-93c3-9c73ea1c5768" containerName="revoke-keys" Sep 16 14:10:17.188489 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.188343 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2698b1b4-3481-41fa-8afc-79551dee1bc8" containerName="istio-proxy" Sep 16 14:10:17.188489 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.188351 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="15c02d88-9bbc-4fea-93c3-9c73ea1c5768" containerName="revoke-keys" Sep 16 14:10:17.188489 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.188357 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2698b1b4-3481-41fa-8afc-79551dee1bc8" containerName="istio-proxy" Sep 16 14:10:17.191276 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.191259 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:17.194067 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.194035 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-6361a7e3-kserve-self-signed-certs\"" Sep 16 14:10:17.216523 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.216498 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx"] Sep 16 14:10:17.275743 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.275693 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:10:17.298424 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.298399 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-model-cache\") pod \"e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:17.298544 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.298439 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4262af8-9b10-4d80-96cf-8f8de161c2a6-tls-certs\") pod \"e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:17.298544 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.298465 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-tmp-dir\") pod \"e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:17.298657 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.298540 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl6kf\" (UniqueName: \"kubernetes.io/projected/a4262af8-9b10-4d80-96cf-8f8de161c2a6-kube-api-access-pl6kf\") pod \"e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:17.298657 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.298626 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-dshm\") pod \"e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:17.298791 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.298659 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-home\") pod \"e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:17.399736 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.399695 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-dshm\") pod \"e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:17.399736 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.399737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-home\") pod \"e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:17.399992 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.399771 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-model-cache\") pod \"e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:17.399992 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.399813 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4262af8-9b10-4d80-96cf-8f8de161c2a6-tls-certs\") pod \"e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:17.399992 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.399850 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-tmp-dir\") pod \"e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:17.399992 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.399895 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl6kf\" (UniqueName: \"kubernetes.io/projected/a4262af8-9b10-4d80-96cf-8f8de161c2a6-kube-api-access-pl6kf\") pod \"e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:17.400202 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.400179 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-model-cache\") pod \"e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:17.400264 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.400202 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-home\") pod \"e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:17.400264 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.400243 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-tmp-dir\") pod \"e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:17.402119 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.402099 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-dshm\") pod \"e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:17.402352 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.402335 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4262af8-9b10-4d80-96cf-8f8de161c2a6-tls-certs\") pod \"e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:17.409393 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.409371 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl6kf\" (UniqueName: \"kubernetes.io/projected/a4262af8-9b10-4d80-96cf-8f8de161c2a6-kube-api-access-pl6kf\") pod \"e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:17.500923 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.500890 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:17.776312 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:17.776265 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx"] Sep 16 14:10:17.779229 ip-10-0-132-9 kubenswrapper[2580]: W0916 14:10:17.779186 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4262af8_9b10_4d80_96cf_8f8de161c2a6.slice/crio-2ce36b35d20b2b48f02427fa1cee836dd6c83618a277dee93d0db594b97a9880 WatchSource:0}: Error finding container 2ce36b35d20b2b48f02427fa1cee836dd6c83618a277dee93d0db594b97a9880: Status 404 returned error can't find the container with id 2ce36b35d20b2b48f02427fa1cee836dd6c83618a277dee93d0db594b97a9880 Sep 16 14:10:18.052463 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:18.052370 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" event={"ID":"a4262af8-9b10-4d80-96cf-8f8de161c2a6","Type":"ContainerStarted","Data":"5185e5aad25a9b48ee58c4fb0d082fddfa52918529b7c978f5bfb1a99ba38d5b"} Sep 16 14:10:18.052463 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:18.052411 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" event={"ID":"a4262af8-9b10-4d80-96cf-8f8de161c2a6","Type":"ContainerStarted","Data":"2ce36b35d20b2b48f02427fa1cee836dd6c83618a277dee93d0db594b97a9880"} Sep 16 14:10:18.074404 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:18.074350 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" podStartSLOduration=1.074333933 podStartE2EDuration="1.074333933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:10:18.073093626 +0000 UTC m=+2366.173906341" watchObservedRunningTime="2026-09-16 14:10:18.074333933 +0000 UTC m=+2366.175146647" Sep 16 14:10:19.006796 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:19.006759 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:10:26.269778 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.269740 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx"] Sep 16 14:10:26.273691 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.273667 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:26.276504 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.276480 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-6361a7e3-kserve-self-signed-certs\"" Sep 16 14:10:26.283759 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.283732 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx"] Sep 16 14:10:26.317644 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.317619 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:10:26.375457 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.375429 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426c2da3-bad3-4404-a724-64165ec80c85-tls-certs\") pod \"e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:26.375601 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.375476 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-dshm\") pod \"e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:26.375601 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.375498 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-home\") pod \"e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:26.375601 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.375560 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-model-cache\") pod \"e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:26.375739 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.375655 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv54l\" (UniqueName: \"kubernetes.io/projected/426c2da3-bad3-4404-a724-64165ec80c85-kube-api-access-hv54l\") pod \"e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:26.375739 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.375683 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-tmp-dir\") pod \"e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:26.476808 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.476769 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-dshm\") pod \"e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:26.476808 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.476817 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-home\") pod \"e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:26.477051 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.476833 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-model-cache\") pod \"e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:26.477051 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.476859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv54l\" (UniqueName: \"kubernetes.io/projected/426c2da3-bad3-4404-a724-64165ec80c85-kube-api-access-hv54l\") pod \"e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:26.477051 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.476881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-tmp-dir\") pod \"e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:26.477051 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.476948 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426c2da3-bad3-4404-a724-64165ec80c85-tls-certs\") pod \"e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:26.477296 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.477271 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-model-cache\") pod \"e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:26.477362 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.477313 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-home\") pod \"e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:26.477362 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.477355 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-tmp-dir\") pod \"e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:26.479210 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.479187 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-dshm\") pod \"e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:26.479598 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.479579 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426c2da3-bad3-4404-a724-64165ec80c85-tls-certs\") pod \"e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:26.486256 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.486233 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv54l\" (UniqueName: \"kubernetes.io/projected/426c2da3-bad3-4404-a724-64165ec80c85-kube-api-access-hv54l\") pod \"e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:26.584404 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.584310 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:26.861244 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:26.861209 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx"] Sep 16 14:10:26.864115 ip-10-0-132-9 kubenswrapper[2580]: W0916 14:10:26.864077 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod426c2da3_bad3_4404_a724_64165ec80c85.slice/crio-5b24f1d82cba3ba864b5569eaa15e503b719474e9858a47eb666f90b395bc6d7 WatchSource:0}: Error finding container 5b24f1d82cba3ba864b5569eaa15e503b719474e9858a47eb666f90b395bc6d7: Status 404 returned error can't find the container with id 5b24f1d82cba3ba864b5569eaa15e503b719474e9858a47eb666f90b395bc6d7 Sep 16 14:10:27.096381 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:27.096343 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" event={"ID":"426c2da3-bad3-4404-a724-64165ec80c85","Type":"ContainerStarted","Data":"5a94c74e44694b5ce4d32d7bdeff9c8003954b9508a4e8137810daf7da594528"} Sep 16 14:10:27.096381 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:27.096387 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" event={"ID":"426c2da3-bad3-4404-a724-64165ec80c85","Type":"ContainerStarted","Data":"5b24f1d82cba3ba864b5569eaa15e503b719474e9858a47eb666f90b395bc6d7"} Sep 16 14:10:27.117874 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:27.117771 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" podStartSLOduration=1.117755405 podStartE2EDuration="1.117755405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:10:27.11640344 +0000 UTC m=+2375.217216155" watchObservedRunningTime="2026-09-16 14:10:27.117755405 +0000 UTC m=+2375.218568120" Sep 16 14:10:27.501158 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:27.501118 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:27.501547 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:27.501178 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:27.513730 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:27.513705 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:28.112909 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:28.112877 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:10:30.506951 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:30.506916 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:10:36.585053 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:36.585017 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:36.585438 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:36.585074 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:36.597801 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:36.597777 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:37.158969 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:37.158922 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:10:54.909758 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:54.909723 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx"] Sep 16 14:10:54.932989 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:54.932951 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:10:56.754424 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:56.754390 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx"] Sep 16 14:10:56.768548 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:56.768515 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:10:57.007890 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:10:57.007795 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:11:03.858994 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:03.858964 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:11:05.706666 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:05.706623 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:11:09.929086 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:09.929038 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" podUID="a4262af8-9b10-4d80-96cf-8f8de161c2a6" containerName="main" containerID="cri-o://5185e5aad25a9b48ee58c4fb0d082fddfa52918529b7c978f5bfb1a99ba38d5b" gracePeriod=45 Sep 16 14:11:10.169111 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.169086 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:11:10.277144 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.277109 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-dshm\") pod \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " Sep 16 14:11:10.277144 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.277147 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-model-cache\") pod \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " Sep 16 14:11:10.277413 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.277169 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-tmp-dir\") pod \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " Sep 16 14:11:10.277413 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.277183 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-home\") pod \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " Sep 16 14:11:10.277413 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.277261 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl6kf\" (UniqueName: \"kubernetes.io/projected/a4262af8-9b10-4d80-96cf-8f8de161c2a6-kube-api-access-pl6kf\") pod \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " Sep 16 14:11:10.277413 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.277306 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4262af8-9b10-4d80-96cf-8f8de161c2a6-tls-certs\") pod \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\" (UID: \"a4262af8-9b10-4d80-96cf-8f8de161c2a6\") " Sep 16 14:11:10.277658 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.277420 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a4262af8-9b10-4d80-96cf-8f8de161c2a6" (UID: "a4262af8-9b10-4d80-96cf-8f8de161c2a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:11:10.277658 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.277441 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "a4262af8-9b10-4d80-96cf-8f8de161c2a6" (UID: "a4262af8-9b10-4d80-96cf-8f8de161c2a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:11:10.277658 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.277511 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-home" (OuterVolumeSpecName: "home") pod "a4262af8-9b10-4d80-96cf-8f8de161c2a6" (UID: "a4262af8-9b10-4d80-96cf-8f8de161c2a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:11:10.277658 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.277582 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:11:10.277658 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.277646 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:11:10.277838 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.277662 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:11:10.279709 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.279675 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4262af8-9b10-4d80-96cf-8f8de161c2a6-kube-api-access-pl6kf" (OuterVolumeSpecName: "kube-api-access-pl6kf") pod "a4262af8-9b10-4d80-96cf-8f8de161c2a6" (UID: "a4262af8-9b10-4d80-96cf-8f8de161c2a6"). InnerVolumeSpecName "kube-api-access-pl6kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:11:10.279709 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.279701 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-dshm" (OuterVolumeSpecName: "dshm") pod "a4262af8-9b10-4d80-96cf-8f8de161c2a6" (UID: "a4262af8-9b10-4d80-96cf-8f8de161c2a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:11:10.279844 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.279797 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4262af8-9b10-4d80-96cf-8f8de161c2a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a4262af8-9b10-4d80-96cf-8f8de161c2a6" (UID: "a4262af8-9b10-4d80-96cf-8f8de161c2a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:11:10.300667 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.300641 2580 generic.go:356] "Generic (PLEG): container finished" podID="a4262af8-9b10-4d80-96cf-8f8de161c2a6" containerID="5185e5aad25a9b48ee58c4fb0d082fddfa52918529b7c978f5bfb1a99ba38d5b" exitCode=0 Sep 16 14:11:10.300794 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.300679 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" Sep 16 14:11:10.300794 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.300679 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" event={"ID":"a4262af8-9b10-4d80-96cf-8f8de161c2a6","Type":"ContainerDied","Data":"5185e5aad25a9b48ee58c4fb0d082fddfa52918529b7c978f5bfb1a99ba38d5b"} Sep 16 14:11:10.300794 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.300785 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx" event={"ID":"a4262af8-9b10-4d80-96cf-8f8de161c2a6","Type":"ContainerDied","Data":"2ce36b35d20b2b48f02427fa1cee836dd6c83618a277dee93d0db594b97a9880"} Sep 16 14:11:10.300957 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.300808 2580 scope.go:117] "RemoveContainer" containerID="5185e5aad25a9b48ee58c4fb0d082fddfa52918529b7c978f5bfb1a99ba38d5b" Sep 16 14:11:10.309867 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.309850 2580 scope.go:117] "RemoveContainer" containerID="5185e5aad25a9b48ee58c4fb0d082fddfa52918529b7c978f5bfb1a99ba38d5b" Sep 16 14:11:10.310121 ip-10-0-132-9 kubenswrapper[2580]: E0916 14:11:10.310102 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5185e5aad25a9b48ee58c4fb0d082fddfa52918529b7c978f5bfb1a99ba38d5b\": container with ID starting with 5185e5aad25a9b48ee58c4fb0d082fddfa52918529b7c978f5bfb1a99ba38d5b not found: ID does not exist" containerID="5185e5aad25a9b48ee58c4fb0d082fddfa52918529b7c978f5bfb1a99ba38d5b" Sep 16 14:11:10.310203 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.310132 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5185e5aad25a9b48ee58c4fb0d082fddfa52918529b7c978f5bfb1a99ba38d5b"} err="failed to get container status \"5185e5aad25a9b48ee58c4fb0d082fddfa52918529b7c978f5bfb1a99ba38d5b\": rpc error: code = NotFound desc = could not find container \"5185e5aad25a9b48ee58c4fb0d082fddfa52918529b7c978f5bfb1a99ba38d5b\": container with ID starting with 5185e5aad25a9b48ee58c4fb0d082fddfa52918529b7c978f5bfb1a99ba38d5b not found: ID does not exist" Sep 16 14:11:10.323311 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.323285 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx"] Sep 16 14:11:10.331768 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.331740 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-6361a7e3-kserve-7f4fb45dc9-n5dwx"] Sep 16 14:11:10.354774 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.354748 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4262af8-9b10-4d80-96cf-8f8de161c2a6" path="/var/lib/kubelet/pods/a4262af8-9b10-4d80-96cf-8f8de161c2a6/volumes" Sep 16 14:11:10.378318 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.378288 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4262af8-9b10-4d80-96cf-8f8de161c2a6-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:11:10.378318 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.378315 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4262af8-9b10-4d80-96cf-8f8de161c2a6-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:11:10.378485 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:10.378327 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl6kf\" (UniqueName: \"kubernetes.io/projected/a4262af8-9b10-4d80-96cf-8f8de161c2a6-kube-api-access-pl6kf\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:11:11.773712 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:11.773643 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" podUID="426c2da3-bad3-4404-a724-64165ec80c85" containerName="main" containerID="cri-o://5a94c74e44694b5ce4d32d7bdeff9c8003954b9508a4e8137810daf7da594528" gracePeriod=45 Sep 16 14:11:12.025227 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.025163 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:11:12.093427 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.093393 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hv54l\" (UniqueName: \"kubernetes.io/projected/426c2da3-bad3-4404-a724-64165ec80c85-kube-api-access-hv54l\") pod \"426c2da3-bad3-4404-a724-64165ec80c85\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " Sep 16 14:11:12.093657 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.093457 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426c2da3-bad3-4404-a724-64165ec80c85-tls-certs\") pod \"426c2da3-bad3-4404-a724-64165ec80c85\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " Sep 16 14:11:12.093657 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.093501 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-tmp-dir\") pod \"426c2da3-bad3-4404-a724-64165ec80c85\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " Sep 16 14:11:12.093657 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.093520 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-home\") pod \"426c2da3-bad3-4404-a724-64165ec80c85\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " Sep 16 14:11:12.093657 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.093550 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-model-cache\") pod \"426c2da3-bad3-4404-a724-64165ec80c85\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " Sep 16 14:11:12.093657 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.093640 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-dshm\") pod \"426c2da3-bad3-4404-a724-64165ec80c85\" (UID: \"426c2da3-bad3-4404-a724-64165ec80c85\") " Sep 16 14:11:12.093946 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.093813 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "426c2da3-bad3-4404-a724-64165ec80c85" (UID: "426c2da3-bad3-4404-a724-64165ec80c85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:11:12.093946 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.093848 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-home" (OuterVolumeSpecName: "home") pod "426c2da3-bad3-4404-a724-64165ec80c85" (UID: "426c2da3-bad3-4404-a724-64165ec80c85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:11:12.093946 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.093894 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-model-cache" (OuterVolumeSpecName: "model-cache") pod "426c2da3-bad3-4404-a724-64165ec80c85" (UID: "426c2da3-bad3-4404-a724-64165ec80c85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:11:12.094101 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.093949 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-tmp-dir\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:11:12.094101 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.093962 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-home\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:11:12.095810 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.095781 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426c2da3-bad3-4404-a724-64165ec80c85-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "426c2da3-bad3-4404-a724-64165ec80c85" (UID: "426c2da3-bad3-4404-a724-64165ec80c85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:11:12.095930 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.095820 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-dshm" (OuterVolumeSpecName: "dshm") pod "426c2da3-bad3-4404-a724-64165ec80c85" (UID: "426c2da3-bad3-4404-a724-64165ec80c85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:11:12.095930 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.095831 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426c2da3-bad3-4404-a724-64165ec80c85-kube-api-access-hv54l" (OuterVolumeSpecName: "kube-api-access-hv54l") pod "426c2da3-bad3-4404-a724-64165ec80c85" (UID: "426c2da3-bad3-4404-a724-64165ec80c85"). InnerVolumeSpecName "kube-api-access-hv54l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:11:12.194893 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.194857 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-model-cache\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:11:12.194893 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.194887 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426c2da3-bad3-4404-a724-64165ec80c85-dshm\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:11:12.194893 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.194899 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv54l\" (UniqueName: \"kubernetes.io/projected/426c2da3-bad3-4404-a724-64165ec80c85-kube-api-access-hv54l\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:11:12.195140 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.194911 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426c2da3-bad3-4404-a724-64165ec80c85-tls-certs\") on node \"ip-10-0-132-9.ec2.internal\" DevicePath \"\"" Sep 16 14:11:12.312398 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.312308 2580 generic.go:356] "Generic (PLEG): container finished" podID="426c2da3-bad3-4404-a724-64165ec80c85" containerID="5a94c74e44694b5ce4d32d7bdeff9c8003954b9508a4e8137810daf7da594528" exitCode=0 Sep 16 14:11:12.312398 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.312348 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" event={"ID":"426c2da3-bad3-4404-a724-64165ec80c85","Type":"ContainerDied","Data":"5a94c74e44694b5ce4d32d7bdeff9c8003954b9508a4e8137810daf7da594528"} Sep 16 14:11:12.312398 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.312354 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" Sep 16 14:11:12.312398 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.312383 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx" event={"ID":"426c2da3-bad3-4404-a724-64165ec80c85","Type":"ContainerDied","Data":"5b24f1d82cba3ba864b5569eaa15e503b719474e9858a47eb666f90b395bc6d7"} Sep 16 14:11:12.312398 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.312400 2580 scope.go:117] "RemoveContainer" containerID="5a94c74e44694b5ce4d32d7bdeff9c8003954b9508a4e8137810daf7da594528" Sep 16 14:11:12.321478 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.321458 2580 scope.go:117] "RemoveContainer" containerID="5a94c74e44694b5ce4d32d7bdeff9c8003954b9508a4e8137810daf7da594528" Sep 16 14:11:12.321767 ip-10-0-132-9 kubenswrapper[2580]: E0916 14:11:12.321746 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a94c74e44694b5ce4d32d7bdeff9c8003954b9508a4e8137810daf7da594528\": container with ID starting with 5a94c74e44694b5ce4d32d7bdeff9c8003954b9508a4e8137810daf7da594528 not found: ID does not exist" containerID="5a94c74e44694b5ce4d32d7bdeff9c8003954b9508a4e8137810daf7da594528" Sep 16 14:11:12.321834 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.321782 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a94c74e44694b5ce4d32d7bdeff9c8003954b9508a4e8137810daf7da594528"} err="failed to get container status \"5a94c74e44694b5ce4d32d7bdeff9c8003954b9508a4e8137810daf7da594528\": rpc error: code = NotFound desc = could not find container \"5a94c74e44694b5ce4d32d7bdeff9c8003954b9508a4e8137810daf7da594528\": container with ID starting with 5a94c74e44694b5ce4d32d7bdeff9c8003954b9508a4e8137810daf7da594528 not found: ID does not exist" Sep 16 14:11:12.334360 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.334333 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx"] Sep 16 14:11:12.345171 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.342239 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-6361a7e3-kserve-b9fb476dc-c27kx"] Sep 16 14:11:12.353846 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:12.353817 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="426c2da3-bad3-4404-a724-64165ec80c85" path="/var/lib/kubelet/pods/426c2da3-bad3-4404-a724-64165ec80c85/volumes" Sep 16 14:11:13.493259 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:13.493163 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:11:19.997088 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:19.997052 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:11:28.109318 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.109277 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:11:28.426989 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.426895 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-85cfb86b8-5bmzb"] Sep 16 14:11:28.427323 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.427304 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="426c2da3-bad3-4404-a724-64165ec80c85" containerName="main" Sep 16 14:11:28.427419 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.427327 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="426c2da3-bad3-4404-a724-64165ec80c85" containerName="main" Sep 16 14:11:28.427419 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.427345 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4262af8-9b10-4d80-96cf-8f8de161c2a6" containerName="main" Sep 16 14:11:28.427419 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.427354 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4262af8-9b10-4d80-96cf-8f8de161c2a6" containerName="main" Sep 16 14:11:28.427583 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.427440 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4262af8-9b10-4d80-96cf-8f8de161c2a6" containerName="main" Sep 16 14:11:28.427583 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.427452 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="426c2da3-bad3-4404-a724-64165ec80c85" containerName="main" Sep 16 14:11:28.430441 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.430419 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85cfb86b8-5bmzb" Sep 16 14:11:28.434798 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.434779 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 16 14:11:28.434922 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.434832 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 14:11:28.434922 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.434844 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wb4dz\"" Sep 16 14:11:28.434922 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.434880 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 14:11:28.435079 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.434971 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 16 14:11:28.443092 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.443063 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cfb86b8-5bmzb"] Sep 16 14:11:28.536757 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.536721 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfeff8c2-c14a-4604-b23e-bf025796bbc9-maas-api-tls\") pod \"maas-api-85cfb86b8-5bmzb\" (UID: \"bfeff8c2-c14a-4604-b23e-bf025796bbc9\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-5bmzb" Sep 16 14:11:28.536757 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.536756 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfeff8c2-c14a-4604-b23e-bf025796bbc9-metrics-certs\") pod \"maas-api-85cfb86b8-5bmzb\" (UID: \"bfeff8c2-c14a-4604-b23e-bf025796bbc9\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-5bmzb" Sep 16 14:11:28.536984 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.536774 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvfxp\" (UniqueName: \"kubernetes.io/projected/bfeff8c2-c14a-4604-b23e-bf025796bbc9-kube-api-access-qvfxp\") pod \"maas-api-85cfb86b8-5bmzb\" (UID: \"bfeff8c2-c14a-4604-b23e-bf025796bbc9\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-5bmzb" Sep 16 14:11:28.536984 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.536932 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeff8c2-c14a-4604-b23e-bf025796bbc9-openshift-service-ca\") pod \"maas-api-85cfb86b8-5bmzb\" (UID: \"bfeff8c2-c14a-4604-b23e-bf025796bbc9\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-5bmzb" Sep 16 14:11:28.638072 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.638031 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeff8c2-c14a-4604-b23e-bf025796bbc9-openshift-service-ca\") pod \"maas-api-85cfb86b8-5bmzb\" (UID: \"bfeff8c2-c14a-4604-b23e-bf025796bbc9\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-5bmzb" Sep 16 14:11:28.638168 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.638100 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfeff8c2-c14a-4604-b23e-bf025796bbc9-maas-api-tls\") pod \"maas-api-85cfb86b8-5bmzb\" (UID: \"bfeff8c2-c14a-4604-b23e-bf025796bbc9\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-5bmzb" Sep 16 14:11:28.638168 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.638119 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfeff8c2-c14a-4604-b23e-bf025796bbc9-metrics-certs\") pod \"maas-api-85cfb86b8-5bmzb\" (UID: \"bfeff8c2-c14a-4604-b23e-bf025796bbc9\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-5bmzb" Sep 16 14:11:28.638168 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.638136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvfxp\" (UniqueName: \"kubernetes.io/projected/bfeff8c2-c14a-4604-b23e-bf025796bbc9-kube-api-access-qvfxp\") pod \"maas-api-85cfb86b8-5bmzb\" (UID: \"bfeff8c2-c14a-4604-b23e-bf025796bbc9\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-5bmzb" Sep 16 14:11:28.638871 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.638843 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeff8c2-c14a-4604-b23e-bf025796bbc9-openshift-service-ca\") pod \"maas-api-85cfb86b8-5bmzb\" (UID: \"bfeff8c2-c14a-4604-b23e-bf025796bbc9\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-5bmzb" Sep 16 14:11:28.640784 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.640763 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfeff8c2-c14a-4604-b23e-bf025796bbc9-maas-api-tls\") pod \"maas-api-85cfb86b8-5bmzb\" (UID: \"bfeff8c2-c14a-4604-b23e-bf025796bbc9\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-5bmzb" Sep 16 14:11:28.640927 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.640909 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfeff8c2-c14a-4604-b23e-bf025796bbc9-metrics-certs\") pod \"maas-api-85cfb86b8-5bmzb\" (UID: \"bfeff8c2-c14a-4604-b23e-bf025796bbc9\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-5bmzb" Sep 16 14:11:28.661191 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.661165 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvfxp\" (UniqueName: \"kubernetes.io/projected/bfeff8c2-c14a-4604-b23e-bf025796bbc9-kube-api-access-qvfxp\") pod \"maas-api-85cfb86b8-5bmzb\" (UID: \"bfeff8c2-c14a-4604-b23e-bf025796bbc9\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-5bmzb" Sep 16 14:11:28.740472 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:28.740434 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85cfb86b8-5bmzb" Sep 16 14:11:29.016541 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:29.016513 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cfb86b8-5bmzb"] Sep 16 14:11:29.019025 ip-10-0-132-9 kubenswrapper[2580]: W0916 14:11:29.018992 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfeff8c2_c14a_4604_b23e_bf025796bbc9.slice/crio-ec4ad0b2d6ec3998640216d84d1f9bbb1469e066d4d7d29ed3ceaaf3c5e4011b WatchSource:0}: Error finding container ec4ad0b2d6ec3998640216d84d1f9bbb1469e066d4d7d29ed3ceaaf3c5e4011b: Status 404 returned error can't find the container with id ec4ad0b2d6ec3998640216d84d1f9bbb1469e066d4d7d29ed3ceaaf3c5e4011b Sep 16 14:11:29.020392 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:29.020377 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 14:11:29.386924 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:29.386835 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85cfb86b8-5bmzb" event={"ID":"bfeff8c2-c14a-4604-b23e-bf025796bbc9","Type":"ContainerStarted","Data":"ec4ad0b2d6ec3998640216d84d1f9bbb1469e066d4d7d29ed3ceaaf3c5e4011b"} Sep 16 14:11:32.403135 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:32.403096 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85cfb86b8-5bmzb" event={"ID":"bfeff8c2-c14a-4604-b23e-bf025796bbc9","Type":"ContainerStarted","Data":"5410b8b5156321dcd99ce4bc8eaab9b27451bac7821efc6bad230bc8d64b7ec4"} Sep 16 14:11:32.403523 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:32.403235 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-85cfb86b8-5bmzb" Sep 16 14:11:34.812938 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:34.812882 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-85cfb86b8-5bmzb" podStartSLOduration=4.426061611 podStartE2EDuration="6.81286246s" totalImagesPullingTime="2.386800849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:11:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:11:32.43788434 +0000 UTC m=+2440.538697055" watchObservedRunningTime="2026-09-16 14:11:34.81286246 +0000 UTC m=+2442.913675184" Sep 16 14:11:34.813989 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:34.813970 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:11:38.412785 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:38.412753 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-85cfb86b8-5bmzb" Sep 16 14:11:40.414970 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:40.414922 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:11:47.404029 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:47.403995 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f8lqw"] Sep 16 14:11:52.986107 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:52.986074 2580 scope.go:117] "RemoveContainer" containerID="a95a6acb218f668b9d72748b4587a028716383c41cf9c28a47563c8bedeca5e3" Sep 16 14:11:52.995587 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:52.995568 2580 scope.go:117] "RemoveContainer" containerID="772a1a0f755f2c7b2fdbf5554a754ed7ba967bb2c91d08e4f09e423e2811bb99" Sep 16 14:11:57.006419 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:57.006388 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fdfbf9c59-422kd_0c703951-bae3-4dcc-a46d-f283cbc9b038/copy-manifests/0.log" Sep 16 14:11:57.026007 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:57.025969 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fdfbf9c59-422kd_0c703951-bae3-4dcc-a46d-f283cbc9b038/manager/0.log" Sep 16 14:11:57.603501 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:57.603469 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-85cfb86b8-5bmzb_bfeff8c2-c14a-4604-b23e-bf025796bbc9/maas-api/0.log" Sep 16 14:11:57.935634 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:57.935510 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826120-ljkdk_0f9c6684-c951-4ec8-bd65-aff71f653452/cleanup/0.log" Sep 16 14:11:59.189970 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:59.189942 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-th4jv_0f95b231-d809-49a0-bdbd-5b711ab3a87f/kuadrant-console-plugin/0.log" Sep 16 14:11:59.442148 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:11:59.442057 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-f8lqw_c412e0bc-13ed-48c9-b3be-c8350310b2ae/limitador/0.log" Sep 16 14:12:00.273686 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:00.273658 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b66dcfb4c-h7zvs_4807f9b2-4daa-475e-95f8-3c25029887d5/kube-auth-proxy/0.log" Sep 16 14:12:01.413394 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:01.413355 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-xpgql_097529a4-8834-45e1-a5f5-05f80f8157de/main/0.log" Sep 16 14:12:01.522400 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:01.522366 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm8vcn_759b88a0-2f0d-4e7f-bb7c-efe9e7efe470/main/0.log" Sep 16 14:12:01.751103 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:01.751074 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-lr4n9_15242a69-7482-4d28-9e00-388900df6334/main/0.log" Sep 16 14:12:06.847213 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:06.847179 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:11.825714 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:11.825677 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cmvcd_5f469d5f-6b1f-4f0e-910d-bff21be9a372/global-pull-secret-syncer/0.log" Sep 16 14:12:11.927201 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:11.927168 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" Sep 16 14:12:12.060190 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:12.060147 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-9.ec2.internal_4fc4f536cfbc959096a0f31f1c1c0873/haproxy/0.log" Sep 16 14:12:12.422452 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:12.422420 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwcp_4c53ff24-05da-4f65-8508-84dda5a72a7d/konnectivity-agent/0.log" limit=1024 Sep 16 14:12:16.262787 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:16.262761 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-th4jv_0f95b231-d809-49a0-bdbd-5b711ab3a87f/kuadrant-console-plugin/0.log" Sep 16 14:12:16.334041 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:16.334006 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-f8lqw_c412e0bc-13ed-48c9-b3be-c8350310b2ae/limitador/0.log" Sep 16 14:12:17.932664 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:17.932629 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_804ff96f-49e3-4fd7-ade8-808e42744b89/alertmanager/0.log" Sep 16 14:12:17.954210 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:17.954182 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_804ff96f-49e3-4fd7-ade8-808e42744b89/config-reloader/0.log" Sep 16 14:12:17.976451 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:17.976417 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_804ff96f-49e3-4fd7-ade8-808e42744b89/kube-rbac-proxy-web/0.log" Sep 16 14:12:17.996405 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:17.996377 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_804ff96f-49e3-4fd7-ade8-808e42744b89/kube-rbac-proxy/0.log" Sep 16 14:12:18.018210 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:18.018184 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_804ff96f-49e3-4fd7-ade8-808e42744b89/kube-rbac-proxy-metric/0.log" Sep 16 14:12:18.042245 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:18.042217 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_804ff96f-49e3-4fd7-ade8-808e42744b89/prom-label-proxy/0.log" Sep 16 14:12:18.070074 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:18.070042 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_804ff96f-49e3-4fd7-ade8-808e42744b89/init-config-reloader/0.log" Sep 16 14:12:18.145284 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:18.145245 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-q7vzb_557e9f3c-9f2c-45db-8a69-08a49a6465d2/kube-state-metrics/0.log" Sep 16 14:12:18.162672 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:18.162650 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-q7vzb_557e9f3c-9f2c-45db-8a69-08a49a6465d2/kube-rbac-proxy-main/0.log" Sep 16 14:12:18.185180 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:18.185100 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-q7vzb_557e9f3c-9f2c-45db-8a69-08a49a6465d2/kube-rbac-proxy-self/0.log" Sep 16 14:12:18.432983 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:18.432956 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rwsmj_839cb0c4-5760-48c2-87ef-015f82193daf/node-exporter/0.log" Sep 16 14:12:18.455328 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:18.455236 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rwsmj_839cb0c4-5760-48c2-87ef-015f82193daf/kube-rbac-proxy/0.log" Sep 16 14:12:18.475995 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:18.475969 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rwsmj_839cb0c4-5760-48c2-87ef-015f82193daf/init-textfile/0.log" Sep 16 14:12:18.868065 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:18.868018 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68fb5d7d69-ldgsk_da9faeb3-35c3-484c-80a6-7da2accdb5ab/telemeter-client/0.log" Sep 16 14:12:18.898845 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:18.898819 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68fb5d7d69-ldgsk_da9faeb3-35c3-484c-80a6-7da2accdb5ab/reload/0.log" Sep 16 14:12:18.929997 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:18.929967 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68fb5d7d69-ldgsk_da9faeb3-35c3-484c-80a6-7da2accdb5ab/kube-rbac-proxy/0.log" Sep 16 14:12:23.397922 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.397887 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q"] Sep 16 14:12:23.404528 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.404494 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q" Sep 16 14:12:23.406923 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.406899 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-skqhh\"/\"default-dockercfg-x2b6g\"" Sep 16 14:12:23.407061 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.406984 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-skqhh\"/\"openshift-service-ca.crt\"" Sep 16 14:12:23.408143 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.408121 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-skqhh\"/\"kube-root-ca.crt\"" Sep 16 14:12:23.413335 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.413308 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q"] Sep 16 14:12:23.419800 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.419775 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e595945-fd30-4805-aef9-0617feea3b46-lib-modules\") pod \"perf-node-gather-daemonset-8qg9q\" (UID: \"0e595945-fd30-4805-aef9-0617feea3b46\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q" Sep 16 14:12:23.419934 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.419810 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0e595945-fd30-4805-aef9-0617feea3b46-podres\") pod \"perf-node-gather-daemonset-8qg9q\" (UID: \"0e595945-fd30-4805-aef9-0617feea3b46\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q" Sep 16 14:12:23.419934 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.419883 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e595945-fd30-4805-aef9-0617feea3b46-sys\") pod \"perf-node-gather-daemonset-8qg9q\" (UID: \"0e595945-fd30-4805-aef9-0617feea3b46\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q" Sep 16 14:12:23.419934 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.419917 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lvn8\" (UniqueName: \"kubernetes.io/projected/0e595945-fd30-4805-aef9-0617feea3b46-kube-api-access-8lvn8\") pod \"perf-node-gather-daemonset-8qg9q\" (UID: \"0e595945-fd30-4805-aef9-0617feea3b46\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q" Sep 16 14:12:23.420126 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.420066 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0e595945-fd30-4805-aef9-0617feea3b46-proc\") pod \"perf-node-gather-daemonset-8qg9q\" (UID: \"0e595945-fd30-4805-aef9-0617feea3b46\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q" Sep 16 14:12:23.521198 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.521160 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0e595945-fd30-4805-aef9-0617feea3b46-podres\") pod \"perf-node-gather-daemonset-8qg9q\" (UID: \"0e595945-fd30-4805-aef9-0617feea3b46\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q" Sep 16 14:12:23.521373 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.521226 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e595945-fd30-4805-aef9-0617feea3b46-sys\") pod \"perf-node-gather-daemonset-8qg9q\" (UID: \"0e595945-fd30-4805-aef9-0617feea3b46\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q" Sep 16 14:12:23.521373 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.521255 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lvn8\" (UniqueName: \"kubernetes.io/projected/0e595945-fd30-4805-aef9-0617feea3b46-kube-api-access-8lvn8\") pod \"perf-node-gather-daemonset-8qg9q\" (UID: \"0e595945-fd30-4805-aef9-0617feea3b46\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q" Sep 16 14:12:23.521373 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.521304 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0e595945-fd30-4805-aef9-0617feea3b46-proc\") pod \"perf-node-gather-daemonset-8qg9q\" (UID: \"0e595945-fd30-4805-aef9-0617feea3b46\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q" Sep 16 14:12:23.521373 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.521336 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0e595945-fd30-4805-aef9-0617feea3b46-podres\") pod \"perf-node-gather-daemonset-8qg9q\" (UID: \"0e595945-fd30-4805-aef9-0617feea3b46\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q" Sep 16 14:12:23.521373 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.521347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e595945-fd30-4805-aef9-0617feea3b46-lib-modules\") pod \"perf-node-gather-daemonset-8qg9q\" (UID: \"0e595945-fd30-4805-aef9-0617feea3b46\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q" Sep 16 14:12:23.521576 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.521372 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e595945-fd30-4805-aef9-0617feea3b46-sys\") pod \"perf-node-gather-daemonset-8qg9q\" (UID: \"0e595945-fd30-4805-aef9-0617feea3b46\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q" Sep 16 14:12:23.521576 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.521395 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0e595945-fd30-4805-aef9-0617feea3b46-proc\") pod \"perf-node-gather-daemonset-8qg9q\" (UID: \"0e595945-fd30-4805-aef9-0617feea3b46\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q" Sep 16 14:12:23.521576 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.521428 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e595945-fd30-4805-aef9-0617feea3b46-lib-modules\") pod \"perf-node-gather-daemonset-8qg9q\" (UID: \"0e595945-fd30-4805-aef9-0617feea3b46\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q" Sep 16 14:12:23.540375 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.540346 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lvn8\" (UniqueName: \"kubernetes.io/projected/0e595945-fd30-4805-aef9-0617feea3b46-kube-api-access-8lvn8\") pod \"perf-node-gather-daemonset-8qg9q\" (UID: \"0e595945-fd30-4805-aef9-0617feea3b46\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q" Sep 16 14:12:23.722872 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.722838 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q" Sep 16 14:12:23.740624 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.740566 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jb79h_bb307e0c-d166-40bb-ae14-2315eb27a09c/dns/0.log" Sep 16 14:12:23.766159 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.766118 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jb79h_bb307e0c-d166-40bb-ae14-2315eb27a09c/kube-rbac-proxy/0.log" Sep 16 14:12:23.848257 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:23.848226 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mf77j_ab189943-4b62-4e77-87bd-75934dcb19cb/dns-node-resolver/0.log" Sep 16 14:12:24.000179 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:24.000059 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q"] Sep 16 14:12:24.002986 ip-10-0-132-9 kubenswrapper[2580]: W0916 14:12:24.002956 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0e595945_fd30_4805_aef9_0617feea3b46.slice/crio-e63332836f9f2866d481cf67e586103cbfc94742c8e3da630907a5bc41fe0c73 WatchSource:0}: Error finding container e63332836f9f2866d481cf67e586103cbfc94742c8e3da630907a5bc41fe0c73: Status 404 returned error can't find the container with id e63332836f9f2866d481cf67e586103cbfc94742c8e3da630907a5bc41fe0c73 Sep 16 14:12:24.531258 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:24.531229 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mslhp_3b1c10dc-eefa-4040-91e6-c52ce36a7ac5/node-ca/0.log" Sep 16 14:12:24.633246 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:24.633216 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q" event={"ID":"0e595945-fd30-4805-aef9-0617feea3b46","Type":"ContainerStarted","Data":"390379d9b9876fe7607e3833d835867793f4d5174c44bf26bcae7150853f592d"} Sep 16 14:12:24.633423 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:24.633264 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q" event={"ID":"0e595945-fd30-4805-aef9-0617feea3b46","Type":"ContainerStarted","Data":"e63332836f9f2866d481cf67e586103cbfc94742c8e3da630907a5bc41fe0c73"} Sep 16 14:12:24.633423 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:24.633320 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q" Sep 16 14:12:24.651288 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:24.651242 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q" podStartSLOduration=1.651227156 podStartE2EDuration="1.651227156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:12:24.650968136 +0000 UTC m=+2492.751780853" watchObservedRunningTime="2026-09-16 14:12:24.651227156 +0000 UTC m=+2492.752039871" Sep 16 14:12:26.351809 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:26.351769 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b66dcfb4c-h7zvs_4807f9b2-4daa-475e-95f8-3c25029887d5/kube-auth-proxy/0.log" Sep 16 14:12:27.387570 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:27.387537 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fr876_b1b0a620-4bae-4969-916f-3c5019de5654/serve-healthcheck-canary/0.log" Sep 16 14:12:27.944919 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:27.944875 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bwssq_97b3faa7-090c-4ad2-975d-2722eaaa0d51/kube-rbac-proxy/0.log" Sep 16 14:12:27.966588 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:27.966558 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bwssq_97b3faa7-090c-4ad2-975d-2722eaaa0d51/exporter/0.log" Sep 16 14:12:27.987780 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:27.987738 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bwssq_97b3faa7-090c-4ad2-975d-2722eaaa0d51/extractor/0.log" Sep 16 14:12:30.648026 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:30.647993 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-8qg9q" Sep 16 14:12:31.173189 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:31.173155 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fdfbf9c59-422kd_0c703951-bae3-4dcc-a46d-f283cbc9b038/manager/0.log" Sep 16 14:12:31.189191 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:31.189161 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fdfbf9c59-422kd_0c703951-bae3-4dcc-a46d-f283cbc9b038/copy-manifests/0.log" Sep 16 14:12:32.612622 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:32.612571 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b66dcfb4c-h7zvs_4807f9b2-4daa-475e-95f8-3c25029887d5/kube-auth-proxy/0.log" Sep 16 14:12:32.618549 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:32.618521 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8dd669f78-5qbld_66a17b7a-d11b-425c-a11f-3b581af44124/manager/0.log" Sep 16 14:12:35.115791 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:35.115759 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b66dcfb4c-h7zvs_4807f9b2-4daa-475e-95f8-3c25029887d5/kube-auth-proxy/0.log" Sep 16 14:12:37.752905 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:37.752874 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-rr4nd_8e942247-6cb7-4e44-82a6-bc7b9c2cf325/migrator/0.log" Sep 16 14:12:37.774268 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:37.774243 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-rr4nd_8e942247-6cb7-4e44-82a6-bc7b9c2cf325/graceful-termination/0.log" Sep 16 14:12:39.233916 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:39.233878 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-97dc7_d55908b4-7d5e-4a7d-8e5b-b84cbac4d4a2/kube-multus/0.log" Sep 16 14:12:39.561106 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:39.561012 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wml9m_70b7167e-0042-4b50-905c-2cb444c08bc3/kube-multus-additional-cni-plugins/0.log" Sep 16 14:12:39.581164 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:39.581136 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wml9m_70b7167e-0042-4b50-905c-2cb444c08bc3/egress-router-binary-copy/0.log" Sep 16 14:12:39.601856 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:39.601824 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wml9m_70b7167e-0042-4b50-905c-2cb444c08bc3/cni-plugins/0.log" Sep 16 14:12:39.621705 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:39.621669 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wml9m_70b7167e-0042-4b50-905c-2cb444c08bc3/bond-cni-plugin/0.log" Sep 16 14:12:39.643271 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:39.643244 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wml9m_70b7167e-0042-4b50-905c-2cb444c08bc3/routeoverride-cni/0.log" Sep 16 14:12:39.663731 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:39.663703 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wml9m_70b7167e-0042-4b50-905c-2cb444c08bc3/whereabouts-cni-bincopy/0.log" Sep 16 14:12:39.683749 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:39.683718 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wml9m_70b7167e-0042-4b50-905c-2cb444c08bc3/whereabouts-cni/0.log" Sep 16 14:12:39.876776 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:39.876703 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-c4xxk_cd501395-aaf2-4005-95bd-f202a13d61b5/network-metrics-daemon/0.log" Sep 16 14:12:39.894214 ip-10-0-132-9 kubenswrapper[2580]: I0916 14:12:39.894184 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-c4xxk_cd501395-aaf2-4005-95bd-f202a13d61b5/kube-rbac-proxy/0.log"